acceptodds
Under review as a conference paper at ICLR 2027

DepthUMI: Active Depth for Universal Manipulation Interfaces

Abstract

Universal Manipulation Interfaces (UMIs) scale robot learning by allowing people to collect demonstrations beyond the robot workspace. Their portability, however, shifts the bottleneck to sensing: a wrist-contained device must recover precise motion despite occlusions and rapid viewpoint changes, while preserving the local 3D geometry needed by the policy. We present DepthUMI, a handheld interface in which the same learned active-depth source supports both motion recovery and policy observation. Rather than treating these as separate sensing problems, DepthUMI combines wide-field stereo, high-rate inertial sensing, and learned active depth in one calibrated assembly. Wide-field visual–inertial reconstruction estimates the demonstrated trajectory, and a calibrated transfer maps camera motion into gripper action supervision. In parallel, learned near-field geometry constrains reconstruction and supplies metric point-cloud observations for policy learning. A simulation study guides the sensing configuration, followed by real-world evaluation of the resulting system. On an instrumented validation rig evaluated against an external Vive reference, DepthUMI achieves an overall mean wrist-camera translation error below 3 mm without using that reference as estimator input. Across three manipulation tasks spanning articulated pulling, peg insertion, and constrained block extraction, learned geometry matches the strong baseline on the coarse pulling task and yields its clearest gains when local geometric precision is decisive. Together, these results show that a single wrist-contained sensing system can produce accurate action supervision and geometry-aware policy inputs. Project page: https://anonymous-submission-x.github.io/DepthUMI/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot learning, universal manipulation interface, active depth sensing, visual-inertial SLAM, 3D visuomotor policy, imitation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oF5XOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose a system called DEPTHUMI. To address the problem of recovering accurate trajectories and obtaining reliable 3D information under occlusion and rapid camera motion with handheld UMI devices, they provide an active-depth-based UMI system. The method is complete and rigorous, and its effectiveness can be seen directly. This is a good system-level work for UMI. However, from a reader's perspective, the readability is very poor. It is very difficult to understand the entire paper. In addition, the novelty in terms of learning is limited. The manuscript needs to be completely restructured. The current presentation is unlikely to meet ICLR standards. I suggest that the authors not rely on AI-assisted writing. They should remove most of the technical terms from the main text and move the technical and implementation details to the appendix. At the same time, the appendix should also be carefully restructured. It only needs to explain the system clearly. The specific details can be placed in a public repository. If there is no further novelty in learning, I suggest that the authors submit the paper to a top robotics conference or journal after restructuring the manuscript.

Critical strengths
  1. The experiments are complete and rigorous. The evaluation protocol is well standardized, and many engineering details are clearly described. This provides sufficient information for readers to reproduce the work.
  2. Figures 5 and 7 clearly show the advantages of the method. The difference between factory depth and learned depth is obvious.
Critical weaknesses
  1. The readability of the paper is very poor. It makes the paper very difficult to read. The work itself is complete, but the manuscript is not well organized. Some parts read like an experimental report. The logic in some parts of the manuscript is poor, which may be caused by AI-assisted writing. I suggest that the authors further improve the readability of the manuscript. The authors may understand the AI-generated language, but it is very difficult for readers encountering this work for the first time. Terms such as DP and DP3 should be written in full and cited when they first appear, such as Diffusion Policy and 3D Diffusion Policy. Experimental protocols such as RGB DP, DP3–Raw, and DP3–Active appear repeatedly in the paper without being defined. This will confuse readers. Content like Section 3.3 should be placed in the appendix. Readers do not care about data such as the reprojection errors reported there. At the same time, the structure of the appendix is also unclear. Figure 2(b) has very poor readability. It is difficult to understand the conclusion the authors want to convey. The same problem also appears in Figures 5(g) and 5(h).
  2. The main contribution of the method is system integration. I did not find any new contribution in learning. This does not seem to fit the scope of ICLR.
Questions for authors
  1. Judging from the demonstration video, the learning-based depth method seems quite slow. What are its frame rate and latency during deployment?
  2. Have the authors tried changing the projector power? Is the comparison with DP3–Raw fair?
1 comment
AuthorsOpenReview ↗

Response to Reviewer oF5X: Manuscript Reorganization, Empirical Contribution, Chunk Latency & Projector Sweep

We sincerely thank Reviewer oF5X for recognizing that our method and experiments are "complete and rigorous", our evaluation protocol is "well standardized", and our visual comparisons clearly demonstrate the advantage of learned active depth over factory stereo depth.

Above all, we take your critique on readability, organization, and AI-assisted phrasing to heart. Re-reading our initial submission from a fresh reader's view, we fully agree that undefined policy shorthand, low-level calibration metrics in §3, small tracking curves, and repetitive appendix tables made the paper read like an engineering report rather than a clear research paper.

1. Structural & Readability Revision Preserving the Core Storyline (W1)

In the revised PDF (10-page main text), we preserved the exact same research problem, hardware, tasks, and storyline, while reorganizing and rewriting the paper for clarity:

  1. Clear upfront scope & unified terminology: The Abstract and §1 now establish both core bottlenecks upfront—close-range wrist tracking under occlusion and close-range 3D perception where RGB lacks explicit geometry and commercial depth cameras drop out—linking trajectory reconstruction to relative action labels and 3D policy control without unexpected detours. Diffusion Policy (DP), 3D Diffusion Policy (DP3), and all three conditions (DP, DP3-Raw, DP3-Active) are defined before results (§1–§3.4), with internal tags cleaned up across text, tables, and figures.
  2. Concept-driven §3 with engineering tables in App. A–B: Reprojection RMS (Table 7), Kalibr IMU parameters (Table 8), and calibration stages move from §3 to App. A, while pre-hardware simulation moves to App. B (including the former simulation plot, now App. Fig. 8). §3 focuses cleanly on close-range active-stereo failure modes and TCP-anchored 3D control.
  3. Clearer visuals & non-redundant Appendix (App. A–E): Tracking curves (Fig. 4g–h, p.6) are redrawn as pure vectors, the projector sweep visual (App. Fig. 12) is enlarged to full width, and duplicate appendix tables are removed.

2. Empirical Visuomotor Insight for ICLR (W2)

We do not claim a new diffusion architecture or learning objective; rather, DepthUMI contributes a controlled empirical study of close-range 3D representations for wrist-only imitation learning. Under identical demonstrations, action labels, DP3 PointNet+DDIM architecture, 4,096-point FPS sampling, and training budgets (App. C.2 Table 12), feeding factory wrist depth into DP3 (DP3-Raw, 20% [6/30]) underperforms even a 2D RGB baseline (DP, 30% [9/30]) because built-in matching drops ~79% of close-range ROI pixels (8–25 cm) around fingers and thin edges. Replacing raw depth with neural structured-light decoding (DP3-Active) lifts success from 20% to 67% (20/30)—rising from 0% to 70% on Peg Insertion and 10% to 50% on 5 mm-clearance Block Extraction (§4.3 Table 3). This identifies close-range geometric completeness as a key bottleneck in scaling portable wrist interfaces to 3D policies.

3. Chunked Control Timing, Latency & Demo Video Duration (Q1)

  • Why failed DP clips are shorter: In side-by-side rollout videos, DP trials that collided with the peg box or adjacent blocks were halted early for hardware safety, whereas DP3-Active completed the full grasp, mating/extraction, and retraction sequence.
  • Online RTX 5090 compute latency (§4.4 Table 6, App. D.2): Online control executes 6-step (600 ms) action chunks spaced at 10 Hz (100 ms/node) via robot kinematics (no online SLAM). Serial per-chunk compute (excluding camera exposure and arm motion) averages 44.33 ms for DP, 71.28 ms for DP3-Raw, and 146.10 ms (P95 149.51 ms) for DP3-Active—spanning 56.00 ms FP8 depth decoding (~17.9 FPS), 53.84 ms 4,096-point FPS cloud construction, and 35.34 ms DDIM denoising (24.4% of the 600 ms chunk horizon; 104.82 ms / 17.5% with random sampling).

4. IR Projector Power Sweep (Levels 0–6) & Matched DP3-Raw Setup (Q2)

Sweeping all 7 Gemini 335 projector levels under fixed exposure (3,000) and gain (16) (§4.4 Table 5, App. Fig. 12) shows that adjusting projector power alone never eliminates close-range factory holes: Level 0 (off) drops full-frame factory coverage to 19.39% (8.80% close-range ROI), Level 1 (ours) reaches 53.60% full-frame and 20.83% close-range ROI coverage at 1.23% IR saturation, and Levels 2–6 saturate up to 52.57% of close-range pixels while close-range factory coverage stalls at 21.09%–23.15% (~77%–79% missing). Note that 100% for the dense learned decoder reflects output completeness rather than ground-truth accuracy; task-level geometric fidelity is validated in Table 3 under matched Level 1 acquisition and 4,096-point FPS preprocessing.

Reviewer_vA2GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce DepthUMI, a UMI-style handheld gripper that combines a T265 wide-field stereo fisheye camera, a dedicated IMU, and an active IR stereo camera (Orbbec Gemini 335) whose depth is decoded with a frozen learned structured-light model. The main pitch is that the active IR depth serves a dual use: (1) it improves pose tracking during data collection, and (2) it improves task performance when used as a policy input. To evaluate (1), the authors compare real-world tracking accuracy across the factory T265 SDK tracker, ORB-SLAM3 stereo-inertial tracking, and an OKVIS2-X-based stereo fisheye-inertial pipeline further constrained by a depth term (with depth from either the learned IR camera or an offline fisheye-derived reference). To evaluate (2), they compare policy success rates with RGB, factory depth, and learned active depth as policy inputs. Prior systems use separate sensing for these roles: FastUMI tracks with the T265 alone, and Cloud-UMI adds a depth camera for policy observations while still tracking with the T265. DepthUMI instead uses the same depth source for both. The authors also include simulation experiments that incrementally add IMU and then depth to a stereo baseline (stereo → +IMU → +IMU+depth) across left-wrist, right-wrist, and head-mounted camera placements in a bimanual setup, though how the placement results inform the final design (single arm UMI) is not clear. It's also not clear why the data collection system was not able to reduce components (e.g., removing the stereo fisheye camera), given the dual use of the active stereo IR camera.

My evaluation centers on whether the central claim, that a single active depth sensing system can serve a dual use for pose tracking and policy input, is well justified and useful. I refer to the two parts as (1) improved tracking and (2) improved policy performance throughout my review.

Critical strengths

S1: Policy performance (2) is justified through a clean real-robot comparison of policy observations: RGB DP vs. DP3-Raw (factory depth) vs. DP3-Active (learned depth) (Table 3). All conditions share the same demonstrations, pose tracking, action labels, data splits, architecture, and training budget, so only the observation input changes. Learned depth reaches 67% total success vs. 20% for factory depth and 30% for RGB, with the largest gains on the precision tasks. This validates that learned active depth can improve policy performance over both RGB and factory depth from the same camera for precise real-world manipulation. This finding that depth is useful as a policy input is not novel, but it does isolate part (2) of their dual-use claim.

Critical weaknesses

W1: The tracking benefit (1) is not sufficiently justified in the real-world experiments.

(a) Tables 1 and 2 include no ablation of the depth term within the OKVIS2-X pipeline. The gain over ORB-SLAM3 (3.117 → 2.875 mm in Table 1) is therefore confounded between the backend change (ORB-SLAM3 → OKVIS2-X) and the addition of depth. Running OKVIS2-X without the depth term on the same recordings in the real-world would directly address this.

(b) The “Fisheye depth” variant, whose depth comes from the T265 alone (each frame is split into 45 perspective stereo pairs processed by FoundationStereo), achieves essentially the same tracking as learned IR depth (2.873 vs. 2.875 mm in Table 1, with similar results in Table 2). Comparable tracking is therefore achievable without the IR camera at all, so the claim that the IR depth specifically improves tracking is not supported. The compute advantage of learned IR depth matters little here, since tracking runs offline on demonstrations and is not needed at deployment.

(c) Tables 6–9 ablate the depth term only in simulation, with an IMU derived from simulator kinematics and depth predicted from rendered active-IR images (App. E.2) rather than the real sensing pipeline. Also, it's unclear how the bimanual left/right and head placement experiments correspond to findings for the handheld device in the real world.

W2: The real-world policy evaluation uses only 10 trials per task per policy, which is low for the main result and limits statistical confidence. The cup study in Table 2 uses only 5 replays and 5 rollouts per motion source. The direction of the Table 3 result is encouraging, but more trials would make S1 considerably stronger.

W3: The current contribution list reads as a description of properties of their method (wrist-contained sensing, learned active depth, experiments spanning sensing to manipulation) rather than isolating the specific new perspectives or scientific contributions that this paper provides. Highlighting the key novelties of this work would help isolate what specific position and role this paper aims to serve given existing literature in depth sensing for UMI, which is presumably the claim of dual-use of the depth sensor for tracking and policy input.

Questions for authors

Q1: Which camera and IMU does the ORB-SLAM3 baseline in Tables 1 and 2 use? My understanding is that it uses the T265 fisheye pair rather than the IR stereo pair. The T265 has its own internal IMU; is there a reason a dedicated external IMU was used instead?

Q2: Why does the system need both the T265 and the IR stereo camera? If the IR depth camera serves both tracking and policy input, it seems the design could reduce components. The conclusion mentions a future single wider-field RGB–IR stereo device to reduce sensor redundancy; real-world evidence for why both are needed in the current system (e.g., ablate tracking with only the IR camera and IMU) would help. An IR + IMU-only tracking result would directly test whether the dual use reduces the need for the T265, which bears on my main concern (W1).

Q3: How does this dual use of a depth camera relate to iPhone-based handheld grippers such as the Dobb-E Stick [1], whose onboard depth could likewise aid tracking and serve as a policy input?

[1] Shafiullah, Nur Muhammad Mahi, et al. "On bringing robots home." arXiv preprint arXiv:2311.16098 (2023).

1 comment
AuthorsOpenReview ↗

Response to Reviewer vA2G: Tracking Controls, Sensor Complementarity, and Prior Handheld RGB-D Systems

We thank Reviewer vA2G for the rigorous critique and for recognizing our controlled real-robot observation comparison (67% DP3-Active vs. 20% DP3-Raw and 30% DP). Below we present the real-world ablations in the revised manuscript addressing W1(a–c), W2–W3, and Q1–Q3.

1. Same-Backend Depth & Sensor-Modality Ablations (W1a, W1b, Q2)

In §4.4 Table 4 (7 recordings, 35 runs/row, 0.2 s-trimmed inner ATE) and §4.2 Table 2 (30 complete cup demonstrations), we isolate depth, wide-field vision, and inertia directly inside OKVIS2-X:

(a) OKVIS2-X Sensor-Modality Ablation Across 7 Recordings (§4.4 Table 4)

Sparse Visual InputIMUDepth FactorScorable Full Window ↑\uparrowBounded (≤100\le 100 mm) ↑\uparrowMean Inner ATE (mm) ↓\downarrow
Gemini IR stereoExternal HI71None20/352/35669,334.3
Gemini IR stereoExternal HI71Learned IR5/353/352,469.5
T265 fisheye stereoExternal HI71None35/3535/352.932
T265 fisheye stereoExternal HI71Learned IR35/3535/352.911
T265 fisheye stereoNoneNone35/3535/358.017
T265 fisheye stereoNoneLearned IR35/3535/358.665

(b) 30-Episode Cup Supervision & Execution (§4.2 Table 2)

Motion SourceSupportMedian ATE (mm) ↓\downarrowAction RMSE (k=1/16k=1 / 16) ↓\downarrowReplay / Policy ↑\uparrow
ORB-SLAM328/305.1772.83 / 9.30 mm5/5, 4/5
OKVIS2-X (VIO, w/o depth)30/303.1191.53 / 7.41 mm5/5, 4/5
Fisheye depth (45-view)30/302.7821.42 / 7.04 mm5/5, 5/5
Learned IR depth30/302.7231.51 / 7.27 mm5/5, 5/5

Direct answers on fisheye depth & sensor complementarity (W1a, W1b, W3, Q2):

  1. Offline fisheye depth equivalence (W1b): You are right that offline 45-view Fisheye depth achieves matching tracking accuracy (2.873 vs. 2.875 mm full-window ATE in §4.1 Table 1; 2.782 vs. 2.723 mm in Table 2). We do not claim an offline tracking gain unique to active IR when peripheral texture is visible.
  2. Why Gemini 335 and T265 are complementary (Q2): We retain Gemini 335 primarily for close-range (8–25 cm) online 3D point-cloud control, where the FR3 wrist carries only Gemini 335 (no T265) and decodes active IR in 56.00 ms (Table 6)—whereas 45-view FoundationStereo takes seconds/frame and lacks active speckle on textureless contact surfaces. Conversely, Gemini IR + HI71 alone (2/35–3/35 bounded, Table 4) fails under finger/object occlusion and cannot replace wide-field fisheye tracking. Reusing onboard Gemini depth in offline OKVIS2-X captures the 12.7% cup ATE reduction (3.119 -> 2.723 mm) at ~1/50th the offline depth compute.

2. ORB-SLAM3 Inputs & External vs. Internal IMU (Q1)

ORB-SLAM3 in Tables 1–2 uses the exact same T265 fisheye stereo + calibrated 1 kHz external HiPNUC IMU streams as OKVIS2-X; thus most of the gain over ORB-SLAM3 (3.117 -> 2.875 mm in Table 1; 5.177 -> 3.119 mm in Table 2) comes from the OKVIS2-X backend, while depth adds local volumetric regularization (2.932 -> 2.911 mm in Table 4; 3.119 -> 2.723 mm in Table 2). In Kalibr (App. A.2 Table 8), the T265 internal IMU runs at 200.1 Hz gyro / 64.0 Hz accel (200.05 Hz fused) with a 16×16\times–554×554\times higher gyro bias random walk (σbg=1.42×10−6\sigma_{bg} = 1.42{\times}10^{-6} vs. 2.56×10−9 rad/s2/Hz2.56{\times}10^{-9}\text{ rad/s}^2/\sqrt{\text{Hz}} on HI71). Substituting the T265 internal IMU in OKVIS2-X drops bounded runs from 35/35 to 27/35 (w/o depth) and 26/35 (w/ Learned IR), while No IMU stays 35/35 bounded at 8.017 / 8.665 mm (Table 4), supporting our calibrated external IMU configuration.

3. Positioning Relative to Dobb-E, Simulation Role & Trial Scope (W1c, W2, W3 & Q3)

  • Dobb-E (Shafiullah et al., 2023) (W3, Q3): Dobb-E is an important predecessor using iPhone ARKit odometry and dToF LiDAR to train RGB-D 2D feature-map policies (HPR) for household tasks. Without claiming a direct hardware comparison against iPhone LiDAR, DepthUMI studies a distinct regime: close-range (8–25 cm) explicit 3D point-cloud policies (DP3) under tight clearances alongside volumetric bundle adjustment (OKVIS2-X). Under matched DP3 settings, replacing raw wrist depth (20.83% close-range ROI coverage, Table 5) with learned active depth lifts overall success from 20% (6/30) to 67% (20/30)—reaching 70% vs. 0% Raw (10% RGB) on Peg Insertion and 50% vs. 10% Raw/RGB on 5 mm-clearance Block Extraction (Table 3).
  • Simulation & trial scope (W1c, W2): RoboDojo simulation (App. B) guided pre-hardware sensor selection; real-world claims stand on physical ablations over 10 interleaved trials/task (seed 42).
Reviewer_7RgkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DepthUMI is a wrist sensing system for collecting manipulation demonstrations. It combines stereo cameras, an IMU, and learned active IR depth to reconstruct motion and generate point-cloud observations. The calibrated trajectories can then provide relative action targets for policy learning. The experiments evaluate tracking, action-label accuracy, and robot performance with RGB, raw depth, and learned depth. The contribution is mainly the sensing and calibration system and its evaluation, rather than a new policy or estimation algorithm.

Critical strengths

The comparison between DP3-Raw and DP3-Active is great. Table 3 are useful results for tasks where missing or inaccurate near-field geometry can make policy learning fail.

The paper evaluates the system beyond global trajectory error. The cup study measures current-relative action error, which is closer to the supervision actually used by the policy. Learned IR reduces median translation RMSE from 2.83 to 1.51 mm at one step and from 9.30 to 7.27 mm at 16 steps (Table 2). This helps connect the sensing system to imitation learning rather than assuming that lower camera ATE automatically gives better actions.

The paper separates camera tracking, reference alignment, and transfer to the tool frame. It also preserves a common current-TCP reference for the action chunk at deployment. These details matter for a demonstration interface, and I appreciate that the appendix reports reference problems and task uncertainty rather than only the best aggregate numbers.

Critical weaknesses

The tracking comparison does not isolate the benefit of depth. The proposed estimator uses an OKVIS2-X-based backend, whereas the baseline uses ORB-SLAM3. The missing comparison is the same proposed backend with its depth factors disabled. This is particularly relevant because learned IR and fisheye depth are almost tied.

The manipulation experiments are too small to tell how stable the gains are. There is one training seed and ten execution trials per task. The appendix appropriately gives wide intervals; for example, the interval around 7/10 peg successes is roughly 40-89%. I find the peg result promising, but would be less confident in smaller differences such as 8/10 versus 5/10 on the drawer. Repeating the precision tasks with independently trained policies would be more useful than adding more tasks with similarly small trial counts.

It is not yet clear how the two uses of depth contribute to the final system together. Table 3 holds motion labels fixed and changes policy observations, so it supports better observation geometry. The cup study addresses motion-label quality separately. Neither experiment measures their combined effect. The paper should keep those conclusions separate, or provide a small comparison varying observation depth and motion-label source independently. Otherwise, the end-to-end gain can easily be attributed to both components when only one is changing in a particular experiment.

Questions for authors
  1. Can the authors report the same-backend, no-depth result on the seven real recordings? This is the comparison that would most affect my assessment of the tracking claim.

  2. What are the main failure modes for DP3-Raw and DP3-Active? Do the peg and block improvements persist across training seeds?

  3. What are the absolute depth-inference and control latencies, and which reconstruction steps run offline? The comparison with the 45-view offline depth reference does not by itself establish deployment speed.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 7Rgk: Same-Backend Depth Ablation, Role Separation, and Chunked Control Latency

We thank Reviewer 7Rgk for the constructive evaluation and for highlighting our DP3-Raw vs. DP3-Active comparison, current-TCP-relative action metric, and staged calibration chain.

1. Same-Backend (OKVIS2-X) No-Depth Ablation on Real Recordings (W1 & Q1)

Alongside our full-window benchmark in §4.1 Table 1 (Learned IR 2.875 mm vs. Fisheye depth 2.873 mm, ORB-SLAM3 3.117 mm, and T265 SDK 119.223 mm), we isolate OKVIS2-X without depth on identical T265 fisheye + calibrated 1 kHz HiPNUC IMU streams in (a) §4.4 Table 4 across all 7 recordings (35 runs/row, 0.2 s-trimmed inner ATE so all 35 runs are uniformly scorable) and (b) §4.2 Table 2 across all 30 cup demonstrations:

(a) OKVIS2-X Sensor & Depth Ablation Across 7 Recordings (§4.4 Table 4, 35 runs/row)

Sparse Visual InputIMUDepth FactorScorable Full Window ↑\uparrowBounded (≤100\le 100 mm) ↑\uparrowMean Inner ATE (mm) ↓\downarrow
Gemini IR stereoExternal HI71None20/352/35669,334.3
Gemini IR stereoExternal HI71Learned IR5/353/352,469.5
T265 fisheye stereoExternal HI71None35/3535/352.932
T265 fisheye stereoExternal HI71Learned IR35/3535/352.911
T265 fisheye stereoNoneNone35/3535/358.017
T265 fisheye stereoNoneLearned IR35/3535/358.665

(b) 30-Episode Cup Supervision & Execution (§4.2 Table 2, shared OKVIS2-X backend)

Motion SourceSupportMedian ATE (mm) ↓\downarrowAction RMSE k=1/3/6/16k=1 / 3 / 6 / 16 (mm) ↓\downarrowReplay / Policy ↑\uparrow
Vive Reference30/30ref.ref.5/5, 5/5
T265 SDK30/3018.4687.82 / 14.00 / 19.77 / 29.751/5, 3/5
ORB-SLAM328/305.1772.83 / 4.34 / 6.17 / 9.305/5, 4/5
OKVIS2-X (VIO, w/o depth)30/303.1191.53 / 2.93 / 4.52 / 7.415/5, 4/5
Fisheye depth30/302.7821.42 / 2.91 / 4.49 / 7.045/5, 5/5
Learned IR depth30/302.7231.51 / 2.91 / 4.53 / 7.275/5, 5/5

Takeaway: Upgrading ORB-SLAM3 to OKVIS2-X on identical fisheye-inertial streams accounts for most of the global gain on textured sequences (Table 1: 3.117 -> 2.875 mm; Table 2: 5.177 -> 3.119 mm), while adding Learned IR within OKVIS2-X provides conditional volumetric regularization: lowering 7-recording inner ATE from 2.932 to 2.911 mm (Table 4), cutting 30-episode cup median ATE by 12.7% (3.119 -> 2.723 mm), tightening k=1/16k=1/16 RMSE (1.53/7.41 -> 1.51/7.27 mm) and P90 upper-tail drift (App. Fig. 11), and preventing VIO divergence in featureless scenes (2,926.3 -> 27.9 mm in all-white simulation, App. Table 10).

2. Decoupled Roles of Depth, Failure Modes & Evaluation Scope (W2, W3 & Q2)

  • Separating supervision vs. observation (W3): Agreed (§1, §4.2–§4.4). Offline supervision relies on wide-field VIO for continuity with depth as a local surface constraint, while online FR3 control uses arm kinematics for TCP pose (no SLAM). Thus Table 3 (DP3-Active 67% [20/30] vs. DP3-Raw 20% [6/30] under matched demos, labels, PointNet+DDIM, 4,096-pt FPS, and budgets; App. Table 12) cleanly isolates close-range observation geometry.
  • Failure modes (Q2, §4.3): DP (RGB, 30%: 7/10, 1/10, 1/10) misjudges axial depth, closing early above the peg or hitting rims/neighbor blocks. DP3-Raw (20%: 5/10, 0/10, 1/10) loses ~79% of close-range ROI pixels (<25 cm, Table 5) to IR saturation and half-occlusion, erasing the peg tip, hole rim, and 5 mm block gap so rollouts stall or drift. DP3-Active (67%: 8/10, 7/10, 5/10) preserves contiguous contact surfaces, failing mainly at extreme angles or inter-block friction.
  • Seed & trial scope (W2, Q2): Policies use one training seed (seed 42, App. C.2, App. E) and 10 interleaved trials/task (±5 cm,±15∘\pm 5\text{ cm}, \pm 15^\circ). Small gaps (8/10 vs. 7/10 Drawer; 5/5 vs. 4/5 Cup) fall within small-sample uncertainty, whereas precision tasks (7/10 vs. 0/10 Peg; 20/30 vs. 6/30 across matched DP3 models) show a clear geometric margin.

3. Chunked Control Schedule & Compute Latency (Q3)

OKVIS2-X and 45-view FoundationStereo run strictly offline. Online (App. C.2, App. D.2), policies execute 6-step (600 ms) prefixes at 10 Hz (100 ms/node), updating the 3-step history after each acknowledged node and replanning per 6-step chunk. Serial RTX 5090 chunk compute (§4.4 Table 6, excluding camera exposure and arm motion) is:

PolicyInputDepth (FP8)Cloud/RGBIODDIMMean / P95 (ms)600 ms Share
DP224×224224{\times}224 RGBn/a3.670.7039.9644.33 / 45.487.4%
DP3-RawFactory XYZ + FPSn/a34.870.8835.5271.28 / 75.4111.9%
DP3-ActiveLearned XYZ + FPS56.0053.840.9135.34146.10 / 149.5124.4%
DP3-ActiveLearned XYZ + Random56.1312.031.0735.59104.82 / 107.7317.5%

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions and Controlled Real-World Ablations

We thank the Area Chair and Reviewers 7Rgk, vA2G, and oF5X for their thoughtful, constructive feedback. We have uploaded an anonymized revised PDF (10-page main text, 22 pages total) that preserves the exact same research problem, wrist hardware, manipulation tasks, and core storyline while adding targeted real-world ablations and reorganizing the presentation:

  1. Same-Backend Tracking & Sensor-Complementarity Controls (7Rgk W1/Q1, vA2G W1/Q1–Q2; §4.1 Table 1, §4.2 Table 2, §4.4 Table 4, App. A.2 Table 8): Isolated OKVIS2-X without depth across all 7 real recordings (2.932 -> 2.911 mm inner ATE in §4.4 Table 4; 2.875 mm Learned IR vs. 2.873 mm Fisheye depth and 3.117 mm ORB-SLAM3 full-window ATE in §4.1 Table 1) and across all 30 cup demonstrations (3.119 -> 2.723 mm median ATE, -12.7%, §4.2 Table 2), alongside Gemini IR + HI71 (2/35–3/35 bounded), No IMU (8.017 / 8.665 mm), and T265 internal IMU (26/35–27/35 bounded) ablations. Both the paper and responses explicitly separate conditional volumetric regularization in offline tracking from the decisive +47 pp (20% -> 67%, Table 3) gain of learned close-range geometry in online DP3 policy control (7Rgk W3, vA2G W3).
  2. Chunked Control Latency & IR Projector Power Sweep (7Rgk Q3, oF5X Q1–Q2; §4.4 Tables 5–6, App. C.2, App. D.2 Fig. 12): Added stage-wise RTX 5090 compute profiling (146.10 ms mean per 600 ms 6-step chunk, including 56.00 ms FP8 depth decoding) and a 7-level (Levels 0–6) projector sweep showing 77%–79% close-range factory depth holes across all active power settings.
  3. Manuscript Clarity & Streamlined Appendix (oF5X W1; Abstract, §1–§5, Fig. 4g–h, App. A–E): Clarified upfront scope and terminology in the Abstract and §1, moved calibration and simulation tables to App. A–B, redrew Fig. 4g–h as pure vectors, refined the SE(3)SE(3) world-offset derivation (App. C.1 Eq. 5) and step-level history update rule (App. C.2), and removed redundant appendix tables.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.