- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
DepState-JEPA extends V-JEPA 2.1 with separate RGB and depth encoders, metric positional encoding, and a shared latent state, DepState. Shared features are aggregated into the recurrent state, while modality-private features provide visual decoding memory. The same spatial readouts support masked-feature completion and action-conditioned future-feature prediction. Robot post-training adds state transitions, proprioceptive and object-motion supervision, and joint–TCP forward-kinematics consistency. The paper evaluates representation transfer, feature prediction, simulated control, and frozen sim-to-real visual-goal planning. The main contribution is a unified state interface, with the most informative empirical evidence coming from the geometry and future-state-supervision ablations on real manipulation.
Critical strengths
-
The geometric state interface is coherent and has useful controlled evidence on real manipulation. The design connects metric RGB-D observations to a state used for action-conditioned prediction, while retaining separate visual decoding memory. Importantly, the depth-content/XYZ experiment retains two active backbones and the RGB-D teacher supervision.
-
The real-robot results provide evidence about what makes the learned state useful for planning. The models are reported to be trained in simulation and deployed with frozen weights, and the full-task criteria include ordered execution and stable release. Removing future-state matching reduces drawer/cup success, despite retaining future-feature and physical supervision. In contrast, removing private decoding worsens drawer/cup feature-prediction, while real success remains. These results suggest that feature fidelity and control utility do not change in lockstep. The FK controls are also informative. These ablations already help explain several auxiliary objectives.
Critical weaknesses
-
The real-robot RGB-D input provenance and registration require verification because they bear directly on the geometric construction. In the supplementary View 1 videos, RGB and colorized depth exhibit a substantial field-of-view and scale mismatch, affecting the platform, table boundary, and robot arm. Exporting native streams before registration is a plausible explanation, and the corresponding motion is broadly consistent. However, the submission does not establish how these displays relate to the actual registered inputs used for control. This is a consequential uncertainty: if registration occurs before model input, it should be straightforward to document; if the displayed pixel correspondences are used directly, the metric lifting and alignment interpretation would be compromised. I need an auditable raw-to-model-input example to assess this part of the method.
-
The distinctive selective-sharing design is less well established than the benefits of geometry and state supervision. Preserving modality-specific decoding information is a reasonable motivation. The less clear point is whether restricting alignment to the shared subspace, together with the additional separation constraints, is responsible for the reported advantage over a simpler fused state. Ex.5 adds private-feature alignment without reducing the existing shared alignment, changing both alignment scope and total strength; Ex.4 changes both private information access and active decoder capacity (App. E.7.2). These comparisons do not isolate the central mechanism. Cross-method results also mix different representation adaptation, encoder-update policies, and supervision (App. E.3/E.5). Finally, the real-robot CEM objective is described as a terminal latent distance without specifying whether it uses DepState or decoded features, making the private pathway's role in actual candidate scoring unclear. A targeted, matched-strength alignment comparison and an explicit planning objective would substantially improve the interpretation.
Questions for authors
-
RGB-D provenance and registration. Are the supplementary View 1 streams native outputs before registration? Which camera/view actually supplies the controller? Please provide a short continuous segment from one demonstration with raw RGB, metric depth, camera/frame IDs and acquisition timestamps, the relevant intrinsics/extrinsics and depth scale, and the registration/crop/resize procedure. Please also show the corresponding postprocessed RGB-D pair and updated intrinsics actually passed to the model.
-
Real-robot planning objective. Please give the complete CEM cost: the prediction and goal representations, normalization, aggregation over tokens/layers/modalities, and whether private-conditioned decoding participates in candidate scoring. Which learned modules are invoked during each candidate rollout? This definition is necessary to interpret the shared-state and private-decoding results.
-
Selective sharing. Does shared-only alignment still outperform full-space alignment when total alignment strength is held fixed? This is the specific comparison for which the current explanation and Ex.5 leave an ambiguity. A matched comparison, or a narrower interpretation of the existing result, would be more informative than an exhaustive sweep of loss removals. Please also clarify the strongest multimodal baseline's trainable modules and physical supervision during robot adaptation, so that the architectural comparison can be assessed.
8 comments
[Q1 | PART 2/4] CAMERA CALIBRATION AND INTRINSICS
[Q1 | PART 2/4] CAMERA CALIBRATION AND INTRINSICS
3. Camera Calibration and the Appearance of View 2
The difference in appearance between View 1 and View 2 is consistent with their different imaging configurations.
Specifically, the Intel RealSense D405 mounted on the Pika gripper from AgileX Robotics uses the left imager of its stereo depth-sensing module to provide the color stream, rather than a separate, independently positioned RGB camera. Its depth stream is derived from the stereo imaging system. In contrast, the D435 uses separate color and stereo-depth imaging modules.
This hardware distinction is consistent with the raw color and depth visualizations of the D405 in View 2 appearing more spatially aligned than those of the D435 in View 1.
The RealSense Viewer screenshot (https://anonymous.4open.science/r/ICLR-9603-DepStateJEPA-Materials/4-RawCamera-RealsenseViewer/RawCamera-RealsenseViewer.png) also shows this distinction: the D405 highlighted by the red box appears as a single imaging module, whereas the D435 has separate color and depth imaging modules.
Camera Intrinsics
The table below reports the native camera intrinsics and the corresponding intrinsics after depth-to-color registration. All values correspond to the 640 × 480 stream profiles used in our experiments.
| View | Camera | Stream / Configuration | fx | fy | cx | cy |
|---|---|---|---|---|---|---|
| View 1 | D435 | Native Color | 607.01196 | 606.85699 | 333.15720 | 247.44922 |
| View 1 | D435 | Native Depth | 386.72366 | 386.72366 | 317.56564 | 245.77191 |
| View 1 | D435 | Registered Color | 607.01196 | 606.85699 | 333.15720 | 247.44922 |
| View 1 | D435 | Registered Depth (aligned to Color) | 607.01196 | 606.85699 | 333.15720 | 247.44922 |
| View 2 | D405 | Native Color | 390.85461 | 390.24884 | 322.37259 | 234.84363 |
| View 2 | D405 | Native Depth | 388.47577 | 388.47577 | 321.48004 | 235.45406 |
| View 1 | D435 | Policy RGB (384 × 384) | 485.60956 | 485.48560 | 202.52576 | 197.95938 |
| View 1 | D435 | Policy Depth (384 × 384) | 485.60956 | 485.48560 | 202.52576 | 197.95938 |
For View 1, the registered color and depth streams share the color-camera pixel coordinate system and therefore have identical intrinsics. The final two rows report the corresponding intrinsics after the policy's center-cropping and resizing operations. This matrix is stored as K in every policy-input NPZ file.
Note: View 2 is used exclusively for supplementary visualization and is not part of the policy-input pipeline. We therefore report only its native camera intrinsics.
[Q1 | PART 3/4] ADDITIONAL DATA AND IMPLEMENTATION FOR INDEPENDENT VERIFICATION
[Q1 | PART 3/4] ADDITIONAL DATA AND IMPLEMENTATION FOR INDEPENDENT VERIFICATION
4. Additional Raw Data and Implementation for Independent Verification
Regarding the request for original camera recordings, we would like to transparently clarify a data-retention issue.
The original recordings were made directly on the workstation used for policy inference, where available storage capacity was already significantly constrained. A single RGB-plus-depth .bag recording typically occupies approximately 5 GB per minute. After the original recording session, we deleted the corresponding .bag files to reclaim storage space. Therefore, unfortunately, the original .bag files corresponding to the submitted supplementary videos are no longer available.
New Recordings for Independent Verification
To address the reviewer's concern for verifiable evidence, we have performed new recordings and prepared two separate anonymous Hugging Face datasets:
- Raw Camera Recordings: Anonymous Hugging Face Dataset — Raw Camera Recordings.
- Registered Policy Inputs: Anonymous Hugging Face Dataset — Registered Policy Inputs.
The two datasets separately provide the original camera recordings and the corresponding registered RGB-D data used as policy inputs.
Camera calibration information: The
.bagfiles preserve the camera stream information and calibration parameters, which can be inspected directly by opening them in the official Intel RealSense Viewer.
Publicly Available Acquisition and Preprocessing Code
We also provide an anonymous code repository containing the complete acquisition and preprocessing implementation:
Anonymous Repository — RealSense RGB-D Tools: (https://anonymous.4open.science/r/ICLR-9603-DepStateJEPA-Materials/3-RealSense-RGBD-Tools/README.md)
The repository includes:
| Script | Description |
|---|---|
record_realsense_rgbd.py | Camera recording script |
realsense_bag_to_video.py | Raw .bag visualization and video conversion script |
realsense_depstate_policy_input.py | Policy-input acquisition and preprocessing script |
| Camera calibration code | Camera calibration and relevant implementation details |
Together, the provided data and scripts document the acquisition and preprocessing pipeline, including:
- Raw RGB and metric depth.
- Camera/frame IDs and acquisition timestamps.
- Relevant camera intrinsics and extrinsics.
- Depth scale.
- Registration, cropping, and resizing procedures.
For clarity, these are newly acquired recordings provided to facilitate independent verification, rather than recovered copies of the original deleted recordings.
[Q1 | PART 1/4] CAMERA ROLES AND RGB-D PROVENANCE
[Q1 | PART 1/4] CAMERA ROLES AND RGB-D PROVENANCE
We thank the reviewer for raising questions about the real-robot camera setup and RGB-D acquisition pipeline. Below, we clarify the camera roles, distinguish the raw visualizations from the registered policy inputs, and provide the relevant calibration, data, and implementation details.
1. Camera Roles
As specified in Appendix B.1 (line 808), the policy uses only View 1, the fixed third-person Intel RealSense D435 camera, during both training and inference.
View 2 is an Intel RealSense D405 mounted on the Pika gripper from AgileX Robotics. It is recorded solely for supplementary visualization and provides a wrist-mounted view of local depth changes during manipulation.
2. Raw Visualization versus Actual Policy Inputs
We confirm that the RGB-D streams shown in Supplementary View 1 are the original, unregistered camera streams; they are not the registered RGB-D inputs supplied to the policy.
During robot execution, our recording script saves two distinct types of data:
.bagfiles, preserving the unregistered sensor recordings.For the supplementary demonstrations, we visualized the original
.bagrecordings rather than the registered RGB-D frames saved by the inference pipeline.To render the raw depth stream for visualization, we played back and exported the
.bagrecordings using the official Intel RealSense Viewer with theDefaultpreset. This preset displays the raw depth stream as a colorized visualization.The following anoymous screenshot link shows the exact visualization configuration used:
RealSense Viewer — Default Preset Visualization : https://anonymous.4open.science/r/ICLR-9603-DepStateJEPA-Materials/4-RawCamera-RealsenseViewer/RawCamera-RealsenseViewer.png
Accordingly, the apparent spatial misalignment in Supplementary View 1 is attributable to its use of raw, unregistered streams. This visualization choice explains the apparent spatial misalignment and does not reflect the actual inputs used during policy inference.