- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses reconstruction from synchronized multi-camera videos of robotic manipulation. It introduces RoboCT4D to standardize observations and evaluation across simulated and real-world robot datasets. The proposed model, CxT4D, extends OmniX with separate camera and temporal embeddings, alternating attention across cameras at each timestamp and across time within each camera stream. Its inference pipeline combines overlapping temporal windows, pose-based stitching, and fixed-camera anchoring. Experiments on four subsets show improvements in depth estimation and cross-camera rotation accuracy, along with substantial memory savings from windowed inference. The evaluation focuses on depth and camera poses and does not assess dense trajectories of dynamic objects.
Critical strengths
-
Bringing heterogeneous robot recordings into a unified multi-camera temporal interface is useful. The inclusion of simulated and real-world observations, together with stationary and moving cameras, makes the benchmark potentially valuable for studying reconstruction under robot-specific observation patterns. This application and evaluation contribution is a meaningful strength, even if the architectural changes are incremental.
-
The comparisons in Tables 4 and 6 keep model weights and pose post-processing fixed. For inputs spanning eight timestamps, windowing reduces peak memory usage from 19.85 to 9.72 GiB while preserving comparable reconstruction accuracy. This provides relatively clear evidence of an efficiency benefit and constitutes the paper’s strongest controlled result.
Critical weaknesses
-
The separation of synchronized cross-camera interaction from temporal interaction within each camera stream has close architectural precedents. Most notably, DVGT [1] (CVPR 2026, Sec. 3.2) factorizes attention into intra-view processing, cross-view attention at each timestamp, and temporal attention within each view for geometry reconstruction. This closely related design warrants explicit discussion and comparison. Earlier work also explores this factorization principle. DynamicStereo [2] (CVPR 2023) uses divided space–stereo–time attention for dynamic stereo reconstruction, while SV4D [3] (2024) organizes images into a view–frame grid with separate view and frame attention. Although their task assumptions differ, these methods demonstrate that separating interactions along spatial, view, and temporal dimensions is well established. Similarly, overlapping chunks followed by geometric alignment are central to VGGT-Long [4] (2025).
The robotic setting, especially independently moving cameras, may motivate a useful adaptation. However, the manuscript does not clearly identify the technical advance beyond incorporating these established principles into OmniX while retaining its prediction heads and training objectives. I therefore view the method as an incremental adaptation that would require particularly compelling empirical evidence to support acceptance.
-
The most informative architectural comparison would hold the OmniX initialization, training data, optimization budget, joint multi-camera inputs, and pose post-processing fixed, varying only the proposed embeddings and attention factorization. It is unclear whether such controls are provided. Appendix B.2 states that matched OmniX and dense fine-tuning controls are unavailable, yet Table 5 appears to report corresponding component results. In addition, the “Base Model” results in Table 5 differ substantially from the OmniX results in Table 2 without a clear explanation. These discrepancies make it difficult to attribute the reported gains to the proposed architecture.
The baseline input protocol is also described inconsistently. Section 5.1 states that external methods process cameras independently, whereas Table 2 states that baselines jointly process all camera–time frames. This distinction directly affects how the cross-camera reconstruction results should be interpreted. The paper also lacks discussion of established joint multi-camera temporal systems such as R3D3 [5] (ICCV 2023). Its known-calibration assumptions would need to be explicitly accounted for in assessing its relevance as a comparison.
Finally, the benchmark evaluates depth and camera poses rather than dynamic object trajectories or scene flow (Sec. 3). These measurements do not directly substantiate improvements in temporal correspondence or consistent object motion. The training mixture also draws from dataset sources used in evaluation, but the episode- and scene-level splits are insufficiently specified to assess generalization. This remains an unresolved evaluation-protocol issue and does not, by itself, indicate data leakage. Taken together, these limitations leave the empirical evidence insufficient to offset the limited methodological novelty.
[1] Sicheng Zuo et al. DVGT: Driving Visual Geometry Transformer. CVPR, 2026.
[2] Nikita Karaev et al. DynamicStereo: Consistent Dynamic Depth from Stereo Videos. CVPR, 2023. arXiv:2305.02296.
[3] Yiming Xie et al. SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency. arXiv:2407.17470, 2024.
[4] Kai Deng et al. VGGT-Long: Chunk it, Loop it, Align it—Pushing VGGT’s Limits on Kilometer-scale Long RGB Sequences. ICRA, 2026. First released as arXiv:2507.16443 in 2025.
[5] Aron Schmied et al. R3D3: Dense 3D Reconstruction of Dynamic Scenes from Multiple Cameras. ICCV, 2023. arXiv:2308.14713.
Questions for authors
-
How does CxT4D’s attention factorization differ substantively from DVGT’s cross-view/cross-frame design? What specific mechanism addresses independently moving robot cameras beyond adapting this factorization to OmniX?
-
Were the baselines in Table 2 evaluated on each camera independently or on all cameras jointly? Please resolve the discrepancy between the table caption and Sec. 5.1. If cameras were processed independently, how were coordinate systems and relative scales aligned for cross-camera pose evaluation?
-
Please reconcile Appendix B.2 with Table 5 and explain the discrepancy between Table 5’s “Base Model” and Table 2’s OmniX results. Which comparisons match training data, optimization budgets, camera context, and post-processing? Under these controls, what improvements can be attributed specifically to the proposed embeddings or attention factorization?
-
What are the exact training, validation, and test splits for each RoboCT4D subset? Are the episodes and scenes used for the reported evaluation excluded from both fine-tuning and checkpoint selection?
-
Beyond depth and camera-pose accuracy, is there quantitative evidence of improved dynamic-region reconstruction, temporal correspondence, or 3D object trajectories? Such results would materially affect my assessment of the claimed contribution.
2 comments
Thank you to the authors for their response. The clarifications address the baseline setup, model variants, and data splits, but my main concerns about methodological novelty and attribution of the reported gains remain.
The method remains an incremental adaptation of established view–time factorization principles. The component study still does not sufficiently isolate the contributions of the embeddings and factored attention under matched training and evaluation settings. The added DexYCB results address per-timestamp moving-object geometry, but do not establish improved temporal correspondence or motion consistency. I therefore maintain my original rating.
W1: We agree that view–time factorization and chunk alignment have precedents. We’ll discuss DVGT, DynamicStereo, SV4D, and VGGT-Long explicitly. Our contribution is RoboCT4D and a structured adaptation of OmniX for synchronized robot observations. Q1 clarifies the implementation differences; we do not claim a dedicated motion-conditioned attention mechanism.
W2: We agree that the comparisons need clearer qualification. Q2–Q4 clarify joint inputs, checkpoints, controls, and held-out splits. The component study does not fully isolate factorization; Table 6 isolates chunking. The added DexYCB test supports moving-object geometry, but does not establish trajectory accuracy or temporal consistency. We’ll clarify this scope and discuss R3D3 with its known-calibration assumptions.
Q1: Both methods separate camera and temporal interactions, with these differences:
CxT4D also uses separate camera/time embeddings. With C cameras, T timestamps, and N tokens per image, cross-camera and temporal attention use T(CN)2 and C(TN)2 query–key pairs. On a shared backbone at equal stage count:
This counts cross-axis attention only, not total runtime or memory. We’ll add this calculation and its scope.
When fixed cameras are available, their poses provide one correction per timestamp, applied to all cameras. Moving-camera smoothing stays off. AgiBot-World has no fixed cameras, so anchoring is disabled. Conditioning attention on time-varying relative poses remains future work.
Q2: All Table 2 baselines receive the same joint camera–time layout with flattened image sequence as input. We use released baseline checkpoints and their method-specific preprocessing and we’ll clarify these details. In Table2, we evaluate the baselines using their released checkpoints and official inference procedures, becaus training code is not available across most compared methods (Any4D, V-DPM, and DepthAnythingV3), and VGGT4D is designed as a training-free method. Table 2 therefore compares overall system performance under different training histories. We additionally provide the same-data dense OmniX baseline to assess performance after adaptation to our training data.
Q3: Table 2 uses released OmniX without RoboCT4D fine-tuning. Table 5’s Base Model uses the OmniX architecture, trained on our data with jointly flattened inputs and dense attention. FurnitureBench results are:
The encoding row adds embeddings and continued training together. Both C×T variants start from the same Base Model checkpoint and use joint inputs. This is our closest comparison for factorization, but isolating its contribution requires matching the remaining training and evaluation settings. Table 6 isolates chunking with identical weights, input samples, and pose post-processing. We’ll clarify these distinctions in Appendix B.2.
Q4: Each subset uses a 90%/10% episode-level train/test split, with episodes and scene configurations kept separate. Task categories may overlap.
We used the last checkpoint (epoch 1), without a validation set or test-based checkpoint selection. We’ll document the split in the dataset description.
Q5: RoboCT4D lacks object-motion ground truth, so we ran a small zero-shot DexYCB test. Both models received the same synchronized RGB clips with frozen checkpoints. We identified moving objects using annotated poses and evaluated their masked depth and visible surfaces using sensor depth and shared camera calibration.
These results support better moving-object geometry at each timestamp.