acceptodds
Under review as a conference paper at ICLR 2027

CameraTime Factorization for Unified Multi-Camera 4D Reconstruction

Abstract

Multi-camera 4D reconstruction is important for embodied perception. Synchronized robot observations naturally form a cameratime structure. Cameras at the same timestamp provide complementary geometric views, while observations from the same camera capture temporal evolution. Existing methods often flatten these observations into a single sequence or process each camera independently, and thus do not explicitly model these two relations. In this paper, we first introduce a CameraTime Formulation for embodied 4D reconstruction. According to formulation, we build RoboCT4D, a unified benchmark that standardizes cameratime observations and evaluation across simulated and real robot manipulation datasets. We then propose CxT4D with CameraTime Factorization. It encodes camera and temporal identities separately and alternates global attention across synchronized cameras and along individual camera streams. This design explicitly models cross-camera geometry and temporal correspondence. We further propose a Long-Sequence MV4D Strategy that reduces memory cost while preserving reconstruction accuracy. Experiments on four RoboCT4D subsets show that CxT4D improves depth and cross-camera rotation accuracy over existing methods. Our long-sequence inference reduces peak memory by 51.0% at eight timestamps and 60.3% at twelve timestamps, and enables longer inputs that otherwise run out of memory.

Primary area:
applications to robotics, autonomy, planning
Keywords:
4D reconstruction, computer vision

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gbMJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses reconstruction from synchronized multi-camera videos of robotic manipulation. It introduces RoboCT4D to standardize observations and evaluation across simulated and real-world robot datasets. The proposed model, CxT4D, extends OmniX with separate camera and temporal embeddings, alternating attention across cameras at each timestamp and across time within each camera stream. Its inference pipeline combines overlapping temporal windows, pose-based stitching, and fixed-camera anchoring. Experiments on four subsets show improvements in depth estimation and cross-camera rotation accuracy, along with substantial memory savings from windowed inference. The evaluation focuses on depth and camera poses and does not assess dense trajectories of dynamic objects.

Critical strengths
  1. Bringing heterogeneous robot recordings into a unified multi-camera temporal interface is useful. The inclusion of simulated and real-world observations, together with stationary and moving cameras, makes the benchmark potentially valuable for studying reconstruction under robot-specific observation patterns. This application and evaluation contribution is a meaningful strength, even if the architectural changes are incremental.

  2. The comparisons in Tables 4 and 6 keep model weights and pose post-processing fixed. For inputs spanning eight timestamps, windowing reduces peak memory usage from 19.85 to 9.72 GiB while preserving comparable reconstruction accuracy. This provides relatively clear evidence of an efficiency benefit and constitutes the paper’s strongest controlled result.

Critical weaknesses
  1. The separation of synchronized cross-camera interaction from temporal interaction within each camera stream has close architectural precedents. Most notably, DVGT [1] (CVPR 2026, Sec. 3.2) factorizes attention into intra-view processing, cross-view attention at each timestamp, and temporal attention within each view for geometry reconstruction. This closely related design warrants explicit discussion and comparison. Earlier work also explores this factorization principle. DynamicStereo [2] (CVPR 2023) uses divided space–stereo–time attention for dynamic stereo reconstruction, while SV4D [3] (2024) organizes images into a view–frame grid with separate view and frame attention. Although their task assumptions differ, these methods demonstrate that separating interactions along spatial, view, and temporal dimensions is well established. Similarly, overlapping chunks followed by geometric alignment are central to VGGT-Long [4] (2025).

    The robotic setting, especially independently moving cameras, may motivate a useful adaptation. However, the manuscript does not clearly identify the technical advance beyond incorporating these established principles into OmniX while retaining its prediction heads and training objectives. I therefore view the method as an incremental adaptation that would require particularly compelling empirical evidence to support acceptance.

  2. The most informative architectural comparison would hold the OmniX initialization, training data, optimization budget, joint multi-camera inputs, and pose post-processing fixed, varying only the proposed embeddings and attention factorization. It is unclear whether such controls are provided. Appendix B.2 states that matched OmniX and dense fine-tuning controls are unavailable, yet Table 5 appears to report corresponding component results. In addition, the “Base Model” results in Table 5 differ substantially from the OmniX results in Table 2 without a clear explanation. These discrepancies make it difficult to attribute the reported gains to the proposed architecture.

    The baseline input protocol is also described inconsistently. Section 5.1 states that external methods process cameras independently, whereas Table 2 states that baselines jointly process all camera–time frames. This distinction directly affects how the cross-camera reconstruction results should be interpreted. The paper also lacks discussion of established joint multi-camera temporal systems such as R3D3 [5] (ICCV 2023). Its known-calibration assumptions would need to be explicitly accounted for in assessing its relevance as a comparison.

    Finally, the benchmark evaluates depth and camera poses rather than dynamic object trajectories or scene flow (Sec. 3). These measurements do not directly substantiate improvements in temporal correspondence or consistent object motion. The training mixture also draws from dataset sources used in evaluation, but the episode- and scene-level splits are insufficiently specified to assess generalization. This remains an unresolved evaluation-protocol issue and does not, by itself, indicate data leakage. Taken together, these limitations leave the empirical evidence insufficient to offset the limited methodological novelty.

[1] Sicheng Zuo et al. DVGT: Driving Visual Geometry Transformer. CVPR, 2026.

[2] Nikita Karaev et al. DynamicStereo: Consistent Dynamic Depth from Stereo Videos. CVPR, 2023. arXiv:2305.02296.

[3] Yiming Xie et al. SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency. arXiv:2407.17470, 2024.

[4] Kai Deng et al. VGGT-Long: Chunk it, Loop it, Align it—Pushing VGGT’s Limits on Kilometer-scale Long RGB Sequences. ICRA, 2026. First released as arXiv:2507.16443 in 2025.

[5] Aron Schmied et al. R3D3: Dense 3D Reconstruction of Dynamic Scenes from Multiple Cameras. ICCV, 2023. arXiv:2308.14713.

Questions for authors
  1. How does CxT4D’s attention factorization differ substantively from DVGT’s cross-view/cross-frame design? What specific mechanism addresses independently moving robot cameras beyond adapting this factorization to OmniX?

  2. Were the baselines in Table 2 evaluated on each camera independently or on all cameras jointly? Please resolve the discrepancy between the table caption and Sec. 5.1. If cameras were processed independently, how were coordinate systems and relative scales aligned for cross-camera pose evaluation?

  3. Please reconcile Appendix B.2 with Table 5 and explain the discrepancy between Table 5’s “Base Model” and Table 2’s OmniX results. Which comparisons match training data, optimization budgets, camera context, and post-processing? Under these controls, what improvements can be attributed specifically to the proposed embeddings or attention factorization?

  4. What are the exact training, validation, and test splits for each RoboCT4D subset? Are the episodes and scenes used for the reported evaluation excluded from both fine-tuning and checkpoint selection?

  5. Beyond depth and camera-pose accuracy, is there quantitative evidence of improved dynamic-region reconstruction, temporal correspondence, or 3D object trajectories? Such results would materially affect my assessment of the claimed contribution.

2 comments

W1: We agree that view–time factorization and chunk alignment have precedents. We’ll discuss DVGT, DynamicStereo, SV4D, and VGGT-Long explicitly. Our contribution is RoboCT4D and a structured adaptation of OmniX for synchronized robot observations. Q1 clarifies the implementation differences; we do not claim a dedicated motion-conditioned attention mechanism.

W2: We agree that the comparisons need clearer qualification. Q2–Q4 clarify joint inputs, checkpoints, controls, and held-out splits. The component study does not fully isolate factorization; Table 6 isolates chunking. The added DexYCB test supports moving-object geometry, but does not establish trajectory accuracy or temporal consistency. We’ll clarify this scope and discuss R3D3 with its known-calibration assumptions.

Q1: Both methods separate camera and temporal interactions, with these differences:

AspectDVGTCxT4D
AttentionLocal, cross-view, and cross-frame attention within each blockCross-camera and temporal attention alternate across OmniX global layers
Pose outputOne ego-vehicle pose per timestampOne pose per camera per timestamp

CxT4D also uses separate camera/time embeddings. With CC cameras, TT timestamps, and NN tokens per image, cross-camera and temporal attention use T(CN)2T(CN)^2 and C(TN)2C(TN)^2 query–key pairs. On a shared backbone at equal stage count:

ScheduleCross-axis pairs per stageC=3,T=8C=3,T=8
DVGT-style serialCT(C+T)N2CT(C+T)N^2264N2264N^2
CxT4D, averaged over two stages12CT(C+T)N2\frac{1}{2}CT(C+T)N^2132N2132N^2

This counts cross-axis attention only, not total runtime or memory. We’ll add this calculation and its scope.

When fixed cameras are available, their poses provide one correction per timestamp, applied to all cameras. Moving-camera smoothing stays off. AgiBot-World has no fixed cameras, so anchoring is disabled. Conditioning attention on time-varying relative poses remains future work.

Q2: All Table 2 baselines receive the same joint camera–time layout with flattened image sequence as input. We use released baseline checkpoints and their method-specific preprocessing and we’ll clarify these details. In Table2, we evaluate the baselines using their released checkpoints and official inference procedures, becaus training code is not available across most compared methods (Any4D, V-DPM, and DepthAnythingV3), and VGGT4D is designed as a training-free method. Table 2 therefore compares overall system performance under different training histories. We additionally provide the same-data dense OmniX baseline to assess performance after adaptation to our training data.

Q3: Table 2 uses released OmniX without RoboCT4D fine-tuning. Table 5’s Base Model uses the OmniX architecture, trained on our data with jointly flattened inputs and dense attention. FurnitureBench results are:

Model / variantSI ↓Aff. ↓Rdeg ↓
Released OmniX, Table 20.25440.279858.24°
Base Model (OmniX, same training datasets, wo/ Camera x Time Factorization), Table 50.19910.246327.75°
+ C×T Encoding0.16220.212022.54°
+ Factored Attention0.15880.176819.61°

The encoding row adds embeddings and continued training together. Both C×T variants start from the same Base Model checkpoint and use joint inputs. This is our closest comparison for factorization, but isolating its contribution requires matching the remaining training and evaluation settings. Table 6 isolates chunking with identical weights, input samples, and pose post-processing. We’ll clarify these distinctions in Appendix B.2.

Q4: Each subset uses a 90%/10% episode-level train/test split, with episodes and scene configurations kept separate. Task categories may overlap.

SubsetTest episodes
RobotWin2-4D clean245
RobotWin2-4D randomized245
FurnitureBench12
AgiBot-World69

We used the last checkpoint (epoch 1), without a validation set or test-based checkpoint selection. We’ll document the split in the dataset description.

Q5: RoboCT4D lacks object-motion ground truth, so we ran a small zero-shot DexYCB test. Both models received the same synchronized RGB clips with frozen checkpoints. We identified moving objects using annotated poses and evaluated their masked depth and visible surfaces using sensor depth and shared camera calibration.

ModelMoving-object SI / Aff. ↓Visible-surface Chamfer (cm) ↓
OmniX release0.240 / 0.1492.80
CxT4D0.149 / 0.1332.54

These results support better moving-object geometry at each timestamp.

Thank you to the authors for their response. The clarifications address the baseline setup, model variants, and data splits, but my main concerns about methodological novelty and attribution of the reported gains remain.

The method remains an incremental adaptation of established view–time factorization principles. The component study still does not sufficiently isolate the contributions of the embeddings and factored attention under matched training and evaluation settings. The added DexYCB results address per-timestamp moving-object geometry, but do not establish improved temporal correspondence or motion consistency. I therefore maintain my original rating.

Reviewer_D8zuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper treats synchronized robot cameras as a camera×time grid. It introduces three things:

  • RoboCT4D: four simulated and real subsets under a common protocol.
  • CxT4D: a fine-tuned OmniX with camera and time embeddings and alternating cross-camera and cross-time attention.
  • Windowed long-sequence inference: SE(3) stitching plus fixed-camera anchoring.

Validation covers comparisons with five baselines, a multi- vs single-view study, a memory study, and a FurnitureBench component study.

Critical strengths
  • Clean long-sequence result. With identical weights, chunking cuts peak memory by 51.0% (T=8) and 60.3% (T=12) and enables T=16, with no accuracy loss.
  • Large gains on real data. Rotation error drops from 58.24° to 19.07° on FurnitureBench, and from 10.58° to 2.71° on AgiBot-World, where no anchoring applies.
Critical weaknesses
  • Uncontrolled comparisons. Section 5.1 and the Table 2 caption describe the baseline protocol differently, and preprocessing differs across methods. CxT4D is also fine-tuned on all three benchmark sources with no described held-out split.
  • Factorization is weakly isolated. The only ablation uses 12 episodes and bundles fine-tuning with the embeddings. Factorization alone moves SI only from 0.162 to 0.159 (affine 0.212 to 0.177), with negligible memory change.
Questions for authors
  • Which baseline protocol was used: per-camera, or one joint forward pass?
  • How were evaluation episodes excluded from the fine-tuning data?
  • How does OmniX fine-tuned with dense attention on the same data compare, on all four subsets?
1 comment

W1: All Table 2 baselines receive the same joint camera–time layout with flattened image sequence as input. We use released baseline checkpoints and their method-specific preprocessing and we’ll clarify these details. In Table2, we evaluate the baselines using their released checkpoints and official inference procedures, becaus training code is not available across most compared methods (Any4D, V-DPM, and DepthAnythingV3), and VGGT4D is designed as a training-free method. Table 2 therefore compares overall system performance under different training histories. We additionally provide the same-data dense OmniX baseline to assess performance after adaptation to our training data.

Each subset is split 90%/10% by episode, with episodes and scene configurations kept separate. Task categories may overlap. We use the final-epoch checkpoint, without validation or test-based checkpoint selection. The same-data dense baseline results are provided in Q3.

W2: We agree that the SI improvement is modest and that the 12-episode FurnitureBench study has limited scope. The other metrics show larger changes:

FurnitureBench metric ↓Dense + C×T Encoding+ Factored Attention
SI0.16220.1588
Aff.0.21200.1768
Rdeg22.54°19.61°
Peak memory (GiB)20.0119.85

These results support the design, but are not a fully controlled estimate of factorization alone. The encoding row also includes continued training, so we cannot attribute its gain solely to embeddings. The substantial measured memory saving comes from chunking: with identical weights and pose post-processing, memory drops from 19.85 to 9.72 GiB at T=8, while accuracy remains comparable.

Q1: We ran each baseline in Table 2 on all 24 images in one forward pass (three cameras, eight synchronized timestamps). We'll spell out this input layout in Sec. 5.1.

Q2: We split each subset 9:1 by episode and removed all test episodes from the training data.

Q3: We trained the OmniX-based dense model on the same data. Here is how it compares with CxT4D in Table 2. CxT4D lowers SI and cross-camera rotation error on all four subsets. It lowers Aff. on three subsets, while the dense model has a slightly lower Aff. on RobotWin2-4D clean (0.0138 versus 0.0143).

SubsetDense Base Model (SI / Aff. / Rdeg)CxT4D (SI / Aff. / Rdeg)
RobotWin2-4D clean0.0190 / 0.0138 / 11.88°0.0175 / 0.0143 / 11.83°
RobotWin2-4D randomized0.0227 / 0.0191 / 16.59°0.0216 / 0.0187 / 15.91°
FurnitureBench0.1991 / 0.2463 / 27.75°0.1580 / 0.1761 / 19.07°
AgiBot-World0.0688 / 0.0660 / 4.68°0.0544 / 0.0539 / 2.71°
Reviewer_S48sOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies feed-forward reconstruction from synchronized multi-camera observations of dynamic robot scenes. It introduces RoboCT4D, which combines simulated and real robot recordings under a standardized camera-by-time interface, and adapts OmniX using separate camera and temporal embeddings with alternating attention along these axes. An overlapping-window inference strategy reduces memory consumption for longer inputs. Experiments report improvements in depth and cross-camera rotation accuracy, while temporal pose improvements are less consistent.

Critical strengths
  1. SOTA performace compared with previous feedforward 4D methods.
  2. introducing unified 4d recon into robot scene is reasonable and important.
Critical weaknesses

The experimental protocol is internally inconsistent in ways that prevent a reliable assessment of the architectural contribution. Section 5.1 states that external methods process each camera independently, whereas Table 2 describes a joint forward pass over all camera-by-time frames. Appendix A and Appendix B.2 state that matched component controls are unavailable, but Table 5 contains numerical results for the base model and successive components. Its base-model result also differs substantially from the OmniX result in Table 2. These discrepancies leave unclear which comparisons isolate factorization from joint camera context, fine-tuning, or checkpoint differences.

Questions for authors
  1. For every Table 2 baseline, were the camera streams processed independently or jointly? Please specify the input layout, checkpoint, fine-tuning status, and pose post-processing, and provide a comparison that holds joint camera context and training conditions fixed.
  2. Which experiments produced the populated rows in Table 5, given the statements that matched controls are unavailable? Please explain the difference between its base-model result and the Table 2 OmniX result, and identify which rows genuinely share evaluation settings.
  3. How are training and evaluation episodes separated within RobotWin2-4D, FurnitureBench, and AgiBot-World? Please provide the split identifiers and clarify whether tasks or scene configurations are shared across the split.
  4. Some relevant works regarding 4d recon and robot could be included in related works: DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling; RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments; Thinking in dynamics: How multimodal large language models perceive, track, and reason dynamics in physical 4d world.
  5. What existing evidence supports improved dynamic reconstruction beyond depth and camera pose? Results from the retained trajectory head or a temporal-consistency evaluation would help establish the scope of the 4D claim; otherwise, the claim should be narrowed.
1 comment

W1: Thanks for pointing out these inconsistencies. All Table 2 baselines jointly process flattened image sequence in one forward pass. The per-camera description in Sec. 5.1 was incorrect. The two OmniX-based results also use different checkpoints:

SettingTrainingFurnitureBench SI / Aff. / Rdeg ↓
OmniX, Table 2Released checkpoint; no RoboCT4D fine-tuning0.2544 / 0.2798 / 58.24°
Base Model, Table 5 (OmniX, trained on same training datasets with CxT4D, wo/ Camera x Time Factorization)Dense attention, trained on our data0.1991 / 0.2463 / 27.75°
+ C×T EncodingEmbeddings and continued training0.1622 / 0.2120 / 22.54°
+ Factored AttentionSame starting Base Model checkpoint0.1588 / 0.1768 / 19.61°

The dense and factored C×T variants are our closest architectural comparison, but they do not fully isolate factorization without matching the remaining training and evaluation settings. The final two rows of Table 5, detailed in Table 6, isolate chunking with identical weights, inputs, and pose post-processing. We’ll make these distinctions consistent throughout the paper.

Q1: For Table 2, each external baseline took all flattened image sequence in a single forward pass per episode, using its released checkpoint. VGGT4D also uses its tracker and two-stage refinement. We will align Sec. 5.1 with the Table 2 caption and clarify the checkpoints and pose post-processing.

In Table2, we evaluate the baselines using their released checkpoints and official inference procedures, becaus training code is not available across most compared methods (Any4D, V-DPM, and DepthAnythingV3), and VGGT4D is designed as a training-free method. Table 2 therefore compares overall system performance under different training histories. We additionally provide the same-data dense OmniX baseline to assess performance after adaptation to our training data.

The Base Model uses the OmniX architecture and is trained on the same data as CxT4D. Our closest comparison for factorization is the following pair in Table 5. Both use joint camera inputs, C×T encoding, and the same starting Base Model checkpoint.

VariantSI ↓Aff. ↓Rdeg ↓
Dense + C×T Encoding0.16220.212022.54°
+ Factored Attention0.15880.176819.61°

A fully controlled comparison would also match the remaining training and evaluation settings.

Q2: The Base Model in Table 5 and OmniX in Table 2 use different checkpoints, which explains their different FurnitureBench results:

ModelTrainingSI ↓Aff. ↓Rdeg ↓
OmniX, Table 2Released checkpoint; no RoboCT4D fine-tuning0.25440.279858.24°
Base Model, Table 5Trained on our data with jointly flattened inputs and dense attention0.19910.246327.75°

In Table 5, FT means continued C×T training from the Base Model. The +C×T Encoding and +Factored Attention variants share that starting checkpoint. The final two rows use identical factored weights and pose post-processing. Table 6 details this controlled inference comparison on the same 12 episodes and input samples. We will clarify in Appendix B.2 that the shared-weights statement applies to the final two rows.

Q3: We split each subset separately at the episode level (90% train, 10% test), and kept episodes and scene configurations separate. The same task category can appear in both splits. We used the final-epoch checkpoint without a validation set or test-based selection. We will document the split in the dataset description.

Q4: Thanks for these suggestions. We will cite DynamicVerse for physical-scale multimodal 4D data, RobotWorld for robot task evaluation, and Thinking in Dynamics for spatiotemporal reasoning and dynamic object grounding.

Q5: We also ran a zero-shot test on DexYCB, which captures people manipulating objects by hand. The test measures the geometry of the moving objects. CxT4D reduces visible-surface Chamfer distance from 2.80 to 2.54 cm relative to OmniX. Table 2 measures how well we recover relative camera motion over time. On FurnitureBench, RPEt decreases from 0.0218 to 0.0148 and RPER from 6.24° to 4.57°. We will clarify the scope of both evaluations in the revision.

Comments on the paper

2 comments

Dear Reviewers,

The authors have submitted their rebuttal to address your concerns. As the discussion deadline approaches, please review their response and share any remaining concerns or questions soon so the authors have enough time to respond.

Thank you, -AC

AuthorsOpenReview ↗

Summary of Contributions and Evaluation Clarifications

We thank the reviewers for their comments and suggestions. Our paper contributes RoboCT4D, a unified benchmark for synchronized multi-camera reconstruction in robot scenes, and CxT4D, an OmniX adaptation that explicitly models camera–time structure. Its long-sequence inference strategy reduces peak memory by 51.0% at eight timestamps and 60.3% at twelve timestamps while maintaining comparable accuracy.

For clarity, Table 2 evaluates the released baseline checkpoints with joint multi-camera inputs. Table 5’s Base Model is an OmniX-derived dense model trained on our data using jointly flattened inputs, which explains its different results from released OmniX in Table 2. We use OmniX for the component study because it provides public training code, while complete training pipelines are not uniformly available for the other baselines, and VGGT4D is designed as a training-free method. We additionally report the same-data dense baseline across all four subsets. We also clarified the separation of training and test episodes and scene configurations, expanded the related-work discussion, and added a zero-shot DexYCB test of moving-object geometry.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.