acceptodds
Under review as a conference paper at ICLR 2027

RoGSW4RLD: Feed-Forward 4D Gaussian Lifting for Robot World Model Rollouts

Abstract

Action-conditioned video world models predict future robot interactions from multiple cameras, yet their outputs remain disparate video collections rather than a shared metric scene queryable across viewpoints and time. While existing 4D reconstruction methods offer a path to spatialize these predictions, independently reconstructing and merging each camera stream fails to enforce cross-view consistency. This limitation is particularly detrimental when combining moving robot-mounted cameras with fixed external views. To address this, we introduce RoGSW4RLD, a feed-forward framework that lifts synchronized multi-camera rollouts into a unified, time-queryable metric 4D Gaussian field. Rather than learning a separate geometric transition model, RoGSW4RLD directly reconstructs the visual future generated by existing world models. Its core innovation is a two-stage architecture: Stage 1 jointly forms the metric 4D field by fusing cross-view evidence with robot-specific articulated geometry and kinematics, while Stage 2 refines the field's geometry and appearance while strictly preserving the initial temporal displacements. Evaluated on 256 held-out DROID episodes, RoGSW4RLD significantly outperforms camera-wise reconstruction with calibrated merging, improving novel-view PSNR by 2.15 dB, reducing depth AbsRel by 47%, and lowering robot displacement error by 61%. These robust gains extend to action-conditioned Cosmos 3 rollouts, demonstrating that predicted video futures can be successfully translated into consistent, spatially queryable 4D metric representations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
World Models, Feed-Forward 4D Gaussian Splatting, Multi-Camera Reconstruction, Robot Kinematics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_p42mOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper reconstructs synchronized robot-camera videos, including world-model rollouts, into a shared metric 4D Gaussian field. Stage 1 extends MoVieS with joint cross-view processing, robot-mesh source anchoring, and forward-kinematics-conditioned learned deformation. Stage 2 uses rendering-feedback refinement with position corrections shared across query times, preserving each Gaussian’s center displacement. Evaluation covers recorded observations and Cosmos 3 rollouts from 256 held-out DROID episodes.

Critical strengths

1)The paper addresses a useful problem: converting multi-camera robot videos into a shared metric 4D scene. Combining joint reconstruction with robot geometry and kinematics is well motivated.

2)The method shows substantial improvements over the evaluated baselines. On recorded observations, novel-view PSNR increases from 13.93 to 16.08 dB and depth AbsRel decreases from 0.373 to 0.197 relative to MoVieS. Similar gains on generated rollouts, together with a clearly documented evaluation protocol, support the system’s practical value.

Critical weaknesses

1)The system-level gains and component studies are informative, but the added value over a comparable joint multi-view model remains unclear. The main baselines reconstruct cameras independently, while Table 2 uses inference-time interventions. A comparison under matched training conditions would better establish the contribution of the robot-specific design.

2)The paper reasonably excludes unreliable non-robot motion labels and acknowledges this limitation. Still, robot EPE and whole-image rendering metrics provide limited direct evidence about manipulated-object motion. An evaluation focused on moving objects, even on a small audited subset, would strengthen the evidence for dynamic reconstruction.

Questions for authors
  1. Beyond the inference-time interventions in Table 2, what evidence supports the added value of the robot-specific design over a joint multi-view model trained with comparable data and supervision?

  2. Given the acknowledged limitations of the non-robot motion labels, can you provide evidence focused on manipulated objects, such as results on a small audited subset, to clarify whether the reported gains extend to object dynamics?

3 comments
AuthorsOpenReview ↗

Response to Reviewer p42m

We thank the reviewer for recognizing this problem's value and requesting matched controls and object-motion evidence. RoGSW4RLD makes the predicted interactions of a pretrained video world model accessible as a shared, time-queryable metric 4D Gaussian scene. We combine the world model's visual predictions with feed-forward reconstruction that incorporates robot geometry and kinematics across moving wrist and fixed external views. This connects predicted videos to metric geometry and motion without learning a separate geometric transition model. The new tests establish robot-specific gains beyond matched joint training and extend the reconstruction evidence to manipulated objects.

[W1 & Q1] What establishes robot-specific value beyond a comparably trained joint multi-view model?

Explicit mesh/FK inputs improve robot reconstruction under matched joint training and supervision, and the final two-stage system retains this advantage. We separately trained Joint reconstruction (no mesh/FK inputs) with the same initialization, data, schedule and supervision, including FK targets. It retains joint attention, merged-field losses and calibrated camera trajectories, but omits mesh anchoring and FK-conditioning inputs.

Table R3-1. Matched robot-grounding comparison. 256 clips per input setting; recorded references. PSNR uses robot regions in held-out stereo views; depth/EPE use source-camera robot support. The first two rows isolate mesh/FK inputs at Stage 1; the last adds refinement.

(a) Recorded-video inputs

ModelRobot novel-view PSNR (dB) ↑Robot depth MAE (mm) ↓Robot EPE (mm) ↓
Joint reconstruction (no mesh/FK inputs; Stage 1)18.2828.912.9
Ours (Stage 1)19.4213.48.6
Ours (Stage 1 + Stage 2)19.5913.78.6

(b) Cosmos-generated inputs

ModelRobot novel-view PSNR (dB) ↑Robot depth MAE (mm) ↓Robot EPE (mm) ↓
Joint reconstruction (no mesh/FK inputs; Stage 1)17.3945.813.1
Ours (Stage 1)17.6715.28.7
Ours (Stage 1 + Stage 2)17.5815.58.7

Bold marks the best displayed values within each panel, with ties at reported precision.

The robot-region gains persist beyond joint processing under matched supervision. On recorded inputs, adding mesh/FK inputs at Stage 1 reduces robot depth error by approximately 54% and displacement error by 33%; held-out-view robot PSNR improves by 1.14 dB [95% paired CI: 0.97, 1.31]. All three metrics also improve with generated inputs. The gains cover rendered surface placement, displacement and novel-view appearance, directly supporting robot grounding beyond matched joint training.

The final system retains the advantage: robot depth MAE is 13.7 vs. 28.9 mm on recorded inputs and 15.5 vs. 45.8 mm on Cosmos inputs, alongside better robot-region PSNR and EPE.

[W2 & Q2] Do the gains extend to manipulated-object motion on a checked subset?

Yes. The final model improves object displacement reconstruction over both DROID-adapted and matched independent-camera baselines. For the requested object-specific evaluation, we use released PointWorld trajectories with checked object support and temporal alignment to the fixed window. We selected 30 recorded carried-object clips (16 rigid-object, 14 cloth) using recorded video/state and track consistency, without reconstruction outputs or scores. Projected support was visually checked to remove robot/cable contamination and ambiguous associations. All models share the same retained external-camera trajectories as estimated 3D references.

Table R3-2. Direct manipulated-object evaluation. Recorded inputs; displacement EPE in mm at identical external-camera reference points. The type columns partition the same 30 clips.

ModelAll 30 ↓Rigid 16 ↓Cloth 14 ↓
MoVieS (DROID fine-tuned)24.123.425.0
Independent reconstruction (matched training)20.719.921.7
Ours (Stage 1 + Stage 2)17.417.317.5

The object-motion gain appears in both rigid-object and cloth interactions. The final model lowers mean object EPE by approximately 28% relative to fine-tuned MoVieS and 16% relative to matched independent reconstruction. All methods use the same cameras, trajectories and scored points, providing the requested direct evidence of manipulated-object reconstruction.

Revision. We will add the trained joint control to §5.3 and the object results to §5.2/Appendix C, with cohort selection in B.1 and track alignment, support checks and metric aggregation in B.5.

Thank you for providing the two controlled studies requested in my review.

The matched joint-reconstruction comparison in Table R3-1 is the evidence I was missing. Training a joint model with the same initialization, data, schedule, and supervision, while removing only the mesh anchoring and FK-conditioning inputs, is a fairer control than inference-time interventions. Under this setting, the robot-specific design improves held-out robot-region PSNR and substantially reduces robot depth and displacement error on both recorded and Cosmos-generated inputs. This better establishes the added value of the robot-specific components beyond joint multi-view training.

The audited 30-clip object-motion evaluation in Table R3-2 also helps. Using checked object support, the method reduces object displacement EPE relative to both the DROID-adapted MoVieS baseline and a matched independent-reconstruction model, with gains on both rigid-object and cloth subsets. The subset is still small, but it provides the direct object-centric evidence that was previously lacking.

These additions address the two main weaknesses in my review. I am therefore inclined to raise my rating from 2 (Weak rejection) to 3 (Weak acceptance).

Thank you for your careful reassessment and constructive feedback.

Your suggestions helped us clarify the contribution of the robot-specific components and provide stronger evidence for manipulated-object reconstruction.

We are glad that the resulting matched joint-reconstruction comparison and audited object-motion evaluation have addressed your main concerns, and we sincerely appreciate your time and thoughtful engagement throughout the review process.

Reviewer_HHSeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RoGSW4RLD converts synchronized fixed-camera and wrist-camera videos, including generated rollouts, into a metric, time-queryable 4D Gaussian field. It adapts MoVieS for joint multi-camera processing, anchors robot geometry with a known mesh, conditions learned deformation on forward kinematics, and adds recurrent rendering-feedback refinement. On held-out DROID episodes, the authors report better novel-view rendering, depth, and robot displacement than camera-wise reconstruction baselines. The rollout evaluation uses supplied robot trajectories and compares results with recorded future observations.

Critical strengths

The complete system shows substantial gains in a practically relevant robot setting. On 256 held-out episodes, Table 1 reports novel-view PSNR of 16.08 versus 13.93 dB, depth AbsRel of 0.197 versus 0.373, and robot EPE of 8.6 versus 22.2 mm against the camera-wise MoVieS baseline; the corresponding clip-bootstrap intervals in Table 7 do not overlap. Most gains are already present in Stage 1. This supports the value of metric reconstruction when calibration and robot states are available.

Critical weaknesses
  1. The experiments establish a system-level improvement but do not isolate its methodological source. Table 1 compares the jointly trained, robot-aware method with released baselines that reconstruct each camera separately. This is a reasonable baseline protocol, but joint processing, robot inputs, training, and supervision change together. Table 2 removes components only at inference from a trained checkpoint. Consequently, the gains specifically due to synchronized fusion remain uncertain.
  2. The learned robot-motion mechanism is insufficiently justified against a simpler use of the available kinematics. Given the query-time joint states and link assignments, Eq. (3) provides an analytic FK displacement, while Eqs. (4)–(5) instead predict robot displacement through a learned FK-conditioned path. The formulation states no hard within-link rigidity constraint. In Appendix D.2, analytic FK transport nearly matches novel-view PSNR and slightly improves robot EPE, although robot depth becomes worse. Thus the distinct benefit of learned robot transport remains unclear despite its role as a core method component.
  3. The demonstrated world-model use case relies on supplied future robot states. Section 3 requires joint and gripper states at query times; the Cosmos experiment generates video from an end-effector action chunk but reconstructs it using the recorded future joint trajectory. The novel-view targets are also recorded future views. The reported scores therefore do not separately establish reconstruction quality for a new generated future with only its action chunk available.
Questions for authors
  1. Under matched DROID training data and supervision, how does a MoVieS-style model perform when jointly processing the same synchronized views? Can joint processing and robot priors be evaluated separately?
  2. What does learned FK-conditioned robot deformation provide beyond a hybrid that uses analytic FK for robot Gaussians and learned motion for non-robot elements? Please report within-link rigidity and manipulated-object or contact-region results.
  3. Is di(si)=0d_i(s_i)=0 enforced so that the mesh-anchored point remains the Gaussian’s center at its source time? How are “valid robot-support pixels” selected at inference when a manipulated object occludes a robot link?
  4. For a new Cosmos rollout, how are query-time joint states and wrist-camera poses obtained from the end-effector action chunk? What changes when recorded future states are replaced by controller- or IK-derived trajectories?
3 comments
AuthorsOpenReview ↗

Response to Reviewer HHSe (1/3)

We thank the reviewer for the questions on component contributions, robot transport and rollout inputs. Our contribution connects the visual predictions of an action-conditioned video world model to joint, robot-grounded metric 4D reconstruction. RoGSW4RLD integrates robot geometry and kinematics during feed-forward field formation across moving wrist and fixed external views. The pretrained world model supplies the visual future; our lifter jointly reconstructs its geometry and motion. The added tests isolate component effects and evaluate learned transport at robot-object interfaces.

[W1 & Q1] Under matched training, can joint processing and robot priors be evaluated separately?

Yes. We separately trained controls with the same initialization, DROID clips, schedule, update scheme and supervision, including paired-right and FK targets. All retain calibrated cameras.

Independent reconstruction (matched training) uses separate camera-field losses. Merged-field supervision only supervises their union, without attention or mesh/FK inputs. Joint reconstruction (no mesh/FK inputs) adds cross-camera attention. Ours (no cross-view attention) instead adds mesh/FK inputs. Ours (Stage 1) uses all three. FK training targets remain in these controls.

Table R2-1. Matched controls, recorded inputs/references, 256 clips. Compare Stage 1 rows for component effects; the final row adds Stage 2. Both panels use identical model order.

(a) Novel-view image and metric-depth reconstruction — three held-out stereo cameras. PSNR/LPIPS use the full target images; AbsRel uses valid reference-depth pixels.

ModelPSNR (dB) ↑LPIPS ↓AbsRel ↓
Independent reconstruction (matched training)15.500.4560.220
Merged-field supervision only15.540.3690.196
Joint reconstruction (no mesh/FK inputs)15.570.3640.193
Ours (no cross-view attention; Stage 1)15.420.3700.205
Ours (Stage 1)15.520.3630.200
Ours (Stage 1 + Stage 2)16.080.3710.197

(b) Robot reconstruction. PSNR uses held-out cameras; depth/EPE use the original source-camera robot support. Bold marks best displayed values, including ties.

ModelNovel-view robot PSNR (dB) ↑Source-view robot depth MAE (mm) ↓Source-view robot EPE (mm) ↓
Independent reconstruction (matched training)18.0632.414.7
Merged-field supervision only18.3730.812.3
Joint reconstruction (no mesh/FK inputs)18.2828.912.9
Ours (no cross-view attention; Stage 1)19.3513.48.6
Ours (Stage 1)19.4213.48.6
Ours (Stage 1 + Stage 2)19.5913.78.6

Shared-field supervision improves perceptual quality and depth before mesh/FK inputs are introduced: LPIPS falls 0.456 → 0.369 and AbsRel 0.220 → 0.196, with paired reductions 0.087 [95% clip-bootstrap CI: 0.083, 0.091] and 0.024 [0.018, 0.030]. Adding attention to the robot-aware Stage 1 model improves PSNR by 0.10 dB [0.06, 0.15] and lowers AbsRel 0.205 → 0.200.

Mesh/FK inputs add robot-specific accuracy beyond matched joint training: depth MAE falls 28.9 → 13.4 mm, EPE 12.9 → 8.6 mm, and robot novel-view PSNR rises 1.14 dB [0.97, 1.31]. Refinement then raises full-image PSNR 15.52 → 16.08 dB. The controls establish complementary roles for field formation, robot grounding and refinement.

Table R2-2. Does per-camera fit survive merging? Wrist source-camera PSNR, 256 recorded clips, input/non-input query times pooled. Each checkpoint and query are fixed; only the rendered Gaussian fields change.

ModelOwn-camera PSNR (dB) ↑Calibrated-union PSNR (dB) ↑
Independent reconstruction (matched training)24.6418.51
Merged-field supervision only23.8523.21
Ours (Stage 1)24.6524.12

Almost identical own-camera fit produces different merged scenes. Union supervision recovers most of the merging loss even without attention or mesh/FK inputs, directly supporting shared-field learning.

Revision. We will add the trained-control evidence to §5.3 and its settings/results to A.2/D.1, keeping Table 2 clearly labeled as fixed-checkpoint interventions.

AuthorsOpenReview ↗

Response to Reviewer HHSe (2/3)

[W2 & Q2] What does learned transport provide beyond analytic robot FK? Within-link rigidity and contact-region results.

Contact-region reconstruction. At fixed submitted weights, learned transport improves visual reconstruction at robot-object interfaces compared with the analytic substitution in Appendix D.2. We keep the Stage 1 checkpoint, inputs and bindings fixed and use D.2's robot-transport policy switch; non-robot Gaussians remain identical. This directly addresses the requested contact-region results.

We use the same audited 30 recorded carried-object clips (16 rigid-object, 14 cloth) as the object-motion evaluation, selected using recorded observations and usable tracks rather than reconstruction scores. Robot/object masks are visually checked and reference-defined. For this contact-region evaluation, we use a 3-pixel band around the robot-object image boundary; LPIPS uses a crop around that band. Both policies are scored on identical eligible images at the 12 non-input times in the three source cameras. The secondary generated-reference test retains the 11 clips with valid generated-frame interface masks.

Table R2-3. Contact-region visual reconstruction at robot-object interfaces. Stage 1 only; MSE uses RGB in [0,1].

(a) Recorded inputs and references — 30 clips, 470 eligible images

Robot transportInterface MSE ↓Interface PSNR (dB) ↑Crop LPIPS ↓
Learned (submitted)0.02780918.1750.2714
Analytic substitution0.02835217.9420.2792

(b) Generated inputs and references — 11 clips, 152 eligible images

Robot transportInterface MSE ↓Interface PSNR (dB) ↑Crop LPIPS ↓
Learned (submitted)0.02680318.6170.2996
Analytic substitution0.02699318.3630.3094

On the common 30 recorded clips, interface MSE is lower by 0.000543 [95% paired clip-bootstrap CI: 0.000138, 0.000977]. PSNR improves by 0.233 dB [0.039, 0.436] and crop LPIPS decreases by 0.0079 [0.0042, 0.0117]. On generated clips, crop LPIPS decreases by 0.0099 [0.0040, 0.0166] and PSNR improves by 0.255 dB [0.030, 0.507]; the generated MSE difference is not resolved. Differences use unrounded scores. These are local visual-fidelity gains in the submitted field.

Within-link rigidity is measured on the same 30 recorded clips and Gaussian groups for both policies. We fit a rigid transform from source positions to query centers per link/source image, excluding each group's own source time.

Table R2-4. Within-link rigid-fit residual (mm). Both columns average per-clip means over the same 30 clips. The first weights robot Gaussians equally; the second weights by query-time opacity within each clip. Analytic values are measured, not assumed.

Robot transportMean residualOpacity-weighted mean residual
Learned (submitted)1.720.73
Analytic substitution<0.0001<0.0001

These tests quantify within-link deformation alongside the contact-region visual-reconstruction gain. Learned transport has a millimetre-scale mean residual, while analytic transport is rigid to numerical precision. The local appearance gains complement D.2's lower robot depth error for learned transport (13.4 vs. 15.1 mm at Stage 1).

Revision. We will extend D.2 with these paired contact-region/rigidity results and document the common cohort, masks, eligibility and aggregation in B.5. The interpretation will remain specific to the submitted-checkpoint substitution.

[Q3] Is source-time zero displacement enforced, and how is robot support selected under occlusion?

At Stage 1 it is supervised, not hard-enforced, and measured centers remain close to their mesh anchors. On the original 256-clip diagnostic, the source-time residual has a clip mean of 1.31 mm, pooled median 0.32 mm, and approximate pooled 95th percentile 1.68 mm. These characterize Stage 1; Stage 2 adds the time-shared position correction in Eq. (8).

At inference, mesh depth/link IDs define candidates. We require a common link in a 3 × 3 neighborhood and reject anchoring where predicted scene depth is more than 10 cm in front of the mesh. Rejected pixels keep learned geometry; no measured depth is used. Close-contact occluders can pass this gate. We will specify the inference rule in A.1 and the source-time diagnostic in D.2.

AuthorsOpenReview ↗

Response to Reviewer HHSe (3/3)

[W3 & Q4] Can reconstruction use the action chunk without measured future states, and does it reconstruct the generated future itself?

Yes. We now directly evaluate action-derived reconstruction against the generated frames themselves, for both Stage 1 and the complete two-stage model. Model-side robot inputs come from the same Cosmos action payload and the measured initial state: gripper values are reused from the payload, arm joints are recovered by sequential inverse kinematics using the known robot model, and wrist-camera poses follow from forward kinematics and fixed hand–eye calibration.

Effect on the original evaluation. We first replace recorded future robot inputs while retaining the original recorded-future references.

Table R2-5. Original-reference evaluation after replacing robot inputs. Same Ours (Stage 1 + Stage 2) checkpoint and generated RGB inputs, 256 clips. PSNR uses held-out stereo views; robot depth/EPE use the original source-camera evaluation support. Change = action-derived minus recorded-state inputs. Differences and paired 95% clip-bootstrap intervals use unrounded scores.

MetricRecorded-state inputsAction-derived inputsChange [95% CI]
Novel-view PSNR (dB) ↑15.52215.521−0.001 [−0.002, +0.001]
Robot EPE (mm) ↓8.7228.804+0.082 [+0.057, +0.114]
Robot depth MAE (mm) ↓15.51015.803+0.294 [+0.176, +0.443]

The input replacement has only a small numerical effect: PSNR is unchanged to two decimals, with robot EPE/depth MAE increases of approximately 0.08/0.29 mm.

Combined test: action-derived inputs and generated-frame references. We also reconstruct from generated frames at times {0, 4, 8, 12, 16} and score the other 12 times against the generated frames themselves, using action-derived robot inputs. Recorded-state runs provide paired controls for each stage. This evaluates both requirements together on 256 clips × 12 times × 3 source cameras = 9,216 images.

Table R2-6. Reconstruction of the generated visual future. Full-image metrics at non-input times in the three source-camera streams. These are temporal-interpolation scores, distinct from the novel-view scores in R2-5. The earlier matched-control rows are retained for context.

Reconstruction model / robot inputsPSNR (dB) ↑SSIM ↑LPIPS ↓
MoVieS (DROID fine-tuned)20.5420.7350.281
Independent reconstruction (matched training)19.240.6850.327
Merged-field supervision only20.460.7770.196
Joint reconstruction (no mesh/FK inputs)20.660.7790.190
Ours (Stage 1; recorded states)21.1170.7930.190
Ours (Stage 1; action-derived)21.1220.7930.190
Ours (Stage 1 + Stage 2; recorded states)24.0890.8380.189
Ours (Stage 1 + Stage 2; action-derived)24.0950.8380.189

Generated-video fidelity is retained with action-derived robot inputs. The paired PSNR changes are +0.0049 dB [95% CI: +0.0022, +0.0078] at Stage 1 and +0.0060 dB [+0.0012, +0.0111] for the full model. These sub-0.01 dB mean changes show that reconstruction does not depend materially on replacing action-derived robot inputs with recorded future states on this cohort. The action-derived full model improves generated-reference PSNR over DROID-fine-tuned MoVieS by 3.55 dB [3.41, 3.69], with lower LPIPS (0.189 vs. 0.281).

The references and evaluation protocol are common across input modes; robot-region masks used in auxiliary scoring remain fixed from recorded states. They are evaluation support, not the action-derived robot inputs. The table above reports full-image metrics.

Revision. We will document action decoding, IK, gripper alignment and wrist-pose construction in §5.1/B.3, add the combined generated-reference evaluation for both stages to §5.2/Appendix C, and specify the recorded-reference and generated-reference protocols in B.3/B.5.

Reviewer_1AhkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a feed-forward framework that reconstructs multi-view robot videos into a shared, time-queryable metric 4D Gaussian scene. The method builds upon MoVieS. It first jointly processes multi-view videos with robot geometry and kinematics information to get a 4D Gaussian field, and then further refines the reconstructed geometry and appearance. The proposed method is evaluated on DROID episodes, including recorded observations and Cosmos 3 generated video rollouts, showing improvements in novel-view rendering, depth accuracy, and robot motion reconstruction.

Critical strengths

It is an interesting and important problem to study multi-view 4D reconstruction specifically in robotic settings, which include wrist-view cameras and robot-specific information, and focus more on the robot and manipulated objects than general everyday videos.

Critical weaknesses

Although the problem is specialized for robotic settings, the general problem is still multi-view 4D reconstruction. One concern is that the proposed method is specifically trained on the DROID dataset, but it is unclear whether the other baselines are also fine-tuned on the same robotic data. Therefore, it is unclear whether the comparison is fair.

Also, since the proposed method uses additional robot-specific information, I wonder how much it benefits from this information. Although the paper includes ablations removing robot mesh anchoring and FK conditioning, would a model trained without this information still outperform the baselines?

On the other hand, there is existing work on joint multi-camera 4D reconstruction, such as No Pose, No Problem in 4D: Feed-Forward Dynamic Gaussians from Unposed Multi-View Videos. It would be worthwhile to include comparisons with such methods.

Finally, it would be interesting to know whether the reconstructed 4D scene would be beneficial for robotic tasks. Currently, the paper mainly shows reconstruction performance but does not demonstrate how this reconstruction could be used for robotic tasks. It would be helpful if the authors could discuss this further.

Questions for authors

Please see the weaknesses above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 1Ahk (1/2)

We thank the reviewer for highlighting training fairness, robot information and joint multi-camera reconstruction. RoGSW4RLD bridges action-conditioned video world models and feed-forward 4D Gaussian reconstruction, making predicted robot interactions queryable in a shared metric scene. Moving wrist and fixed external views mix camera motion with scene dynamics, while calibrated merging alone does not enforce shared geometry and motion. We integrate robot geometry and kinematics during joint field formation to reconstruct the frozen world model's predicted future. The new controls show reconstruction gains after DROID adaptation and complementary robot-grounding benefits.

[W1] Are the gains explained by training on DROID while leaving the baselines unadapted?

The reconstruction improvement remains after DROID fine-tuning and under matched-training controls. We fine-tuned MoVieS on the same DROID training clips using its released loss and full-parameter fine-tuning recipe. We also trained Independent reconstruction (matched training) with our initialization, schedule, parameter-update scheme and supervision, including paired-right losses and FK training targets. Both controls reconstruct cameras independently and render their calibrated union.

Table R1-1. DROID-adapted and matched-training controls. Both panels use 256 clips, recorded references and the same three held-out stereo cameras. PSNR/LPIPS use full images; AbsRel uses valid depth-reference pixels. Only the reconstruction input changes.

(a) Recorded-video inputs

ModelPSNR (dB) ↑LPIPS ↓AbsRel ↓
MoVieS (DROID fine-tuned)14.640.4570.236
Independent reconstruction (matched training)15.500.4560.220
Ours (Stage 1)15.520.3630.200
Ours (Stage 1 + Stage 2)16.080.3710.197

(b) Cosmos-generated inputs

ModelPSNR (dB) ↑LPIPS ↓AbsRel ↓
MoVieS (DROID fine-tuned)14.240.4820.259
Independent reconstruction (matched training)14.990.4810.244
Ours (Stage 1)15.040.3940.218
Ours (Stage 1 + Stage 2)15.520.4020.215

The gains persist under stronger controls. The full model gains 1.44 dB [95% paired clip-bootstrap CI: 1.36, 1.53] on recorded inputs and 1.28 dB [1.20, 1.37] on Cosmos inputs over fine-tuned MoVieS. It improves all three metrics over matched independent reconstruction in both settings. Perceptual and depth gains already appear before refinement: Stage 1 lowers recorded-input LPIPS 0.456 → 0.363 and AbsRel 0.220 → 0.200.

[W2] Would a model trained without the additional robot information still outperform the baselines?

Yes. Our joint reconstruction still outperforms both camera-wise baselines in novel-view rendering and depth after removing mesh/FK inputs and supervision. We trained Ours (Stage 1; no mesh/FK inputs or supervision) without mesh anchoring, FK-conditioning inputs, FK targets or FK-derived masks. Calibrated camera poses, including the wrist trajectory, and measured-track supervision remain available.

Table R1-2. Removing mesh/FK inputs and supervision. Recorded inputs/references, 256 clips; identical model order in both panels. Our Stage 1 rows compare removal/restoration of both inputs and supervision; the last row adds refinement.

(a) Image/depth reconstruction in held-out stereo cameras

ModelPSNR (dB) ↑LPIPS ↓AbsRel ↓
MoVieS (DROID fine-tuned)14.640.4570.236
Independent reconstruction (matched training)15.500.4560.220
Ours (Stage 1; no mesh/FK inputs or supervision)15.680.3580.198
Ours (Stage 1)15.520.3630.200
Ours (Stage 1 + Stage 2)16.080.3710.197

(b) Robot reconstruction in source-camera streams

ModelRobot depth MAE (mm) ↓Robot displacement EPE (mm) ↓
MoVieS (DROID fine-tuned)42.915.2
Independent reconstruction (matched training)32.414.7
Ours (Stage 1; no mesh/FK inputs or supervision)48.916.2
Ours (Stage 1)13.48.6
Ours (Stage 1 + Stage 2)13.78.6

Panel (b): depth uses fixed valid robot support; EPE samples rendered 3D displacement at reference-track pixels. Bold marks best displayed values, including ties.

Joint reconstruction still improves every metric in (a) over both camera-wise controls without mesh/FK inputs or supervision. Restoring them adds robot grounding: depth MAE falls 48.9 → 13.4 mm and EPE 16.2 → 8.6 mm at Stage 1. The final system combines this benefit with 16.08 dB PSNR and 0.197 AbsRel.

AuthorsOpenReview ↗

Response to Reviewer 1Ahk (2/2)

[W3] How does the method compare with joint multi-camera reconstruction such as NoPo4D?

The advantage holds with NoPo4D's default temporal pose averaging and with averaging disabled. NoPo4D [1] averages camera-pose predictions over time to enforce its static-rig assumption; disabling this operation retains frame-wise predictions for the moving-wrist setting. Both variants receive the same three source views and five input times as our method.

Table R1-3. Joint multi-view reconstruction with temporal pose averaging on/off. 256 clips per input setting; recorded references throughout. The first three metrics use full images/valid depth pixels in held-out stereo views; robot depth and EPE use the paper's source-camera robot support. NoPo4D uses its released checkpoint without DROID fine-tuning.

(a) Recorded-video inputs

ModelNovel PSNR (dB) ↑Novel LPIPS ↓Novel AbsRel ↓Robot depth MAE (mm) ↓Robot EPE (mm) ↓
NoPo4D — averaging on (default)12.760.4530.286112.128.0
NoPo4D — averaging off13.040.4740.287115.228.1
Ours (Stage 1)15.520.3630.20013.48.6
Ours (Stage 1 + Stage 2)16.080.3710.19713.78.6

(b) Cosmos-generated inputs

ModelNovel PSNR (dB) ↑Novel LPIPS ↓Novel AbsRel ↓Robot depth MAE (mm) ↓Robot EPE (mm) ↓
NoPo4D — averaging on (default)12.600.4740.300136.028.6
NoPo4D — averaging off12.870.4960.303139.328.6
Ours (Stage 1)15.040.3940.21815.28.7
Ours (Stage 1 + Stage 2)15.520.4020.21515.58.7

We express calibrated query cameras relative to NoPo4D's pose estimates and use one source-stereo scale per camera, without target-depth fitting. Both reconstruction stages improve all five metrics over both NoPo4D settings in both input domains. With recorded inputs, the final model reduces robot depth MAE 112.1 → 13.7 mm and EPE 28.0 → 8.6 mm relative to default NoPo4D. These system-level results complement the DROID-adapted and matched-training controls in W1–W2.

[W4] How could the reconstructed scene benefit robotic tasks?

We propose two uses: offline policy supervision and online temporal-context tokens.

Offline policy learning. GaussianDream [2] uses current/future Gaussian reconstruction as VLA training supervision. Our lifter could provide metric geometry and motion pseudo-targets from action-conditioned rollouts of a large pretrained video world model. This would make its predicted interactions available for spatial and motion supervision during policy pretraining or fine-tuning.

Online VLA conditioning. VistaVLA [3] injects compact 3D spatial-semantic tokens constructed from current observations. Our time-queryable field could extend this idea to tokens describing predicted temporal changes. This prospective online use requires lower end-to-end latency and validation of the policy integration.

Revision. We will add these comparisons to §5.2/Appendix C, specify the training and NoPo4D protocols in B.4, and discuss the proposed policy uses in §6.

[1] Balice et al. No Pose, No Problem in 4D: Feed-Forward Dynamic Gaussians from Unposed Multi-View Videos. arXiv:2605.22190, 2026.
[2] Zhang et al. GaussianDream: A Feed-Forward 3D Gaussian World Model for Robotic Manipulation. arXiv:2605.20752, 2026.
[3] Liu et al. VistaVLA: Geometry- and Semantic-Aware 3D Gaussian-Grounded VLA for Robotic Manipulation. arXiv:2607.12356, 2026.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of responses and revised manuscript uploaded

We thank all three reviewers for their thoughtful feedback on our work reconstructing predicted robot interactions from a video world model as a shared, time-queryable metric 4D Gaussian scene.

Revised PDF. We have uploaded the revised pdf with changes marked in blue. Table and section numbers below follow the revision. Table 2 reports separately trained controls. The fixed-checkpoint interventions cited as Table 2 in our earlier response now appear in Appendix D.1 (Table 20).

1. Stronger baselines and matched comparisons (Reviewer 1). Against recorded novel-view images, the full model exceeds DROID-fine-tuned MoVieS by 1.44 dB on recorded inputs and 1.28 dB on generated inputs. It improves novel-view PSNR, LPIPS, and depth AbsRel over matched independent reconstruction in both settings. Both stages outperform released NoPo4D on all five reconstruction metrics with pose averaging on or off. On recorded inputs, Stage 1 trained without mesh/FK inputs or supervision still improves all three novel-view metrics over both camera-wise controls (Tables 1 and 13–16). Section 6 discusses potential uses in policy learning and conditioning.

2. Shared-field learning and robot grounding (Reviewers 2 and 3). Under matched Stage 1 supervision, including FK targets, shared-field supervision reduces LPIPS from 0.456 to 0.369 and depth AbsRel from 0.220 to 0.196 before adding attention or mesh/FK inputs. With robot inputs present, cross-view attention adds 0.10 dB in novel-view PSNR. Adding mesh/FK inputs to the matched joint model reduces robot depth MAE from 28.9 to 13.4 mm and displacement EPE from 12.9 to 8.6 mm (§5.3, Table 2).

3. Generated futures with action-derived inputs (Reviewer 2). We derive robot inputs from the Cosmos action payload and measured initial state, without recorded future robot states. Across 256 clips, we evaluate against generated source-camera frames at the 12 times withheld from reconstruction. The action-derived full model exceeds DROID-fine-tuned MoVieS by 3.55 dB on this temporal-interpolation test. The mean paired change in PSNR from recorded-state to action-derived inputs is +0.0049 dB at Stage 1 and +0.0060 dB for the full model (Table 18).

4. Manipulated-object motion and contact regions (Reviewers 2 and 3). On 30 audited recorded clips, object EPE is 17.4 mm for the full model, 24.1 mm for fine-tuned MoVieS, and 20.7 mm for matched independent reconstruction. Gains hold for both rigid objects and cloth using the same checked PointWorld tracks as estimated references (Table 19). At fixed Stage 1 weights, learned transport improves contact-region PSNR by 0.233 dB over analytic substitution, with a mean within-link rigid-fit residual of 1.72 mm. Analytic transport is rigid to numerical precision and yields slightly lower robot EPE (Tables 24–26).

AuthorsOpenReview ↗

Revised Supplementary Material Uploaded

We have uploaded an updated supplementary video with figures reflecting the quantitative results reported in our responses to the reviewers.

AuthorsOpenReview ↗

Rebuttal Summary for AC

Dear AC,

Thank you for overseeing the review of our submission. We also thank the reviewers for their constructive feedback.

RoGSW4RLD bridges action-conditioned video world models and feed-forward 4D Gaussian reconstruction, turning predicted interactions across moving wrist and fixed external cameras into a shared, time-queryable metric scene.

Strengths Highlighted in the Reviews

  • An important robotics-specific problem (1Ahk, p42m). The reviewers recognized the value of multi-camera metric 4D reconstruction; 1Ahk emphasized wrist cameras, robot-specific information, and the focus on robots and manipulated objects rather than general everyday videos.
  • Well-motivated robot grounding (p42m, HHSe). p42m highlighted combining joint reconstruction with robot geometry and kinematics. HHSe recognized the value of metric reconstruction when calibration and robot states are available.
  • Substantial gains with supporting evidence (HHSe, p42m). On 256 held-out episodes, the camera-wise MoVieS comparison improves PSNR 13.93 → 16.08 dB and AbsRel 0.373 → 0.197. HHSe additionally highlighted robot EPE 22.2 → 8.6 mm, non-overlapping clip-bootstrap intervals, and that most gains already appear in Stage 1.
  • Rollout improvements and clear evaluation (p42m). Similar gains on generated rollouts and a clearly documented evaluation protocol support the system's practical value.

Main Concerns and Responses

1. Training Fairness and Attribution (1Ahk, HHSe, p42m). We added DROID-fine-tuned MoVieS and separately trained controls with matched initialization, data, schedule, and supervision. Shared-field supervision reduces LPIPS 0.456 → 0.369 and AbsRel 0.220 → 0.196 versus independent-field training (R2-1). The full model exceeds fine-tuned MoVieS by 1.44 dB on recorded inputs and 1.28 dB on Cosmos inputs (R1-1). A separately trained variant without mesh/FK inputs or supervision still improves rendering and depth over both camera-wise controls (R1-2).

2. Robot-Specific Value Beyond Joint Training (HHSe, p42m). The joint control retains attention, merged-field losses, and FK supervision, but removes mesh/FK input pathways. Adding these inputs at Stage 1 lowers robot depth MAE 28.9 → 13.4 mm and EPE 12.9 → 8.6 mm, and raises held-out robot-region PSNR by 1.14 dB. All three metrics improve with Cosmos inputs as well (R3-1).

3. Learned Transport, Rigidity, and Anchoring (HHSe). We extended D.2's fixed-checkpoint analytic substitution to the requested contact regions on 30 audited clips. Learned transport improves interface PSNR by 0.233 dB [95% paired clip-bootstrap CI: 0.039, 0.436] and lowers crop LPIPS by 0.0079 [0.0042, 0.0117] (R2-3). Its mean within-link residual is 1.72 mm; analytic transport is rigid to numerical precision (R2-4). For Stage 1 source-time anchoring, we clarified zero-displacement supervision, reported a 1.31 mm mean residual, and specified the predicted-depth inference rule (Q3).

4. Action-Derived Inputs and Generated-Future Fidelity (HHSe). We reuse gripper values from the Cosmos action payload, recover arm joints by sequential IK from the initial state, and derive wrist poses using FK and calibration. On 9,216 held-out-time source-camera images from 256 clips, we evaluate reconstruction against generated frames themselves. Replacing recorded-state inputs changes mean PSNR by less than 0.01 dB at both stages. The action-derived full model reaches 24.095 dB, exceeding fine-tuned MoVieS by 3.55 dB [3.41, 3.69] (R2-6). The original recorded-reference evaluation is also retained (R2-5).

5. Manipulated-Object Motion (p42m). On 30 audited clips (16 rigid-object, 14 cloth), all methods use common checked object support and released PointWorld tracks as estimated 3D references. Final-model object EPE is 17.4 mm, versus 24.1 mm for fine-tuned MoVieS and 20.7 mm for matched independent reconstruction, with gains in both subsets (R3-2). Following the matched joint and object-motion studies, p42m stated that their two main weaknesses were addressed and indicated an inclination to raise their rating from 2 (Weak rejection) to 3 (Weak acceptance).

6. Joint Baselines and Robotic Applications (1Ahk). We evaluated released NoPo4D with temporal pose averaging on/off; both stages outperform both settings across the reported metrics on recorded and Cosmos inputs (R1-3). We also discussed offline policy supervision and prospective online temporal conditioning, separately from the evaluated reconstruction results (W4).

We have incorporated these studies and protocol clarifications into the manuscript as specified in our responses. We hope this summary assists your assessment of the contribution and additional evidence.

Thank you again for your time and consideration.

Best regards,
Authors of Submission 61116

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.