acceptodds
Under review as a conference paper at ICLR 2027

GEARS: Seeing Geometry, Diffusing Actions for Zero-Shot Sim-to-Real Dexterous Manipulation

Abstract

Simulation provides scalable interaction data and privileged geometric labels for dexterous manipulation, yet zero-shot deployment remains limited by coupled visual and physical mismatches. Appearance changes alter the student's visual observations, while hidden simulator states and randomized physical parameters can make multiple expert actions plausible from the same deployable observation. We present Geometric Action Representation for Sim-to-Real (GEARS), a three-stage framework that uses privileged information only during training. GEARS first fine-tunes a visual foundation model with joint inverse-depth and surface-normal prediction and a cross-task consistency objective. This produces a frozen geometric encoder that consumes only RGB at deployment. GEARS then trains a privileged reinforcement learning expert with Virtual Particle Stein Variational Gradient Descent (VP-SVGD), which uses an estimated failure landscape to adapt ten object-pose and physical randomization dimensions. Finally, a diffusion student conditioned on the frozen geometric features and proprioception learns the expert action distribution after hidden simulator information is marginalized. The central design links perception and action modeling: geometric supervision preserves contact-relevant scene structure in the RGB representation, while diffusion represents the action variation induced by unobserved state and physics. Across four chemistry-laboratory tasks on a fixed dual-arm platform, GEARS achieves a 67.5% real-world success rate without real-world policy training, substantially outperforming the strongest controlled baseline at 42.5%, with gains on all four tasks. These results demonstrate the effectiveness of jointly addressing visual domain shift through geometric representation learning and action ambiguity through diffusion-based policy modeling for zero-shot sim-to-real dexterous manipulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
reinforcement learning, visual sim2real

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8m8EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The manuscript mainly addresses the sim-to-real gap for dexterous hands and proposes a three-stage method called GEARS. The method design is reasonable, but the writing is unclear. There are also two suspected AI hallucinations in the paper involving references to the appendix. The most serious problem is that the experiments lack recent methods and classic baselines. The experimental setup is unfair, and maybe there is seed selection. The authors also acknowledge this.

Critical strengths

1.The details of the method are explained thoroughly, so readers can reproduce the method easily.
2.The ablation experiments are sufficient.

Critical weaknesses

1.From a statistical perspective, I question whether the reported positive results may come from the authors selecting random seeds and choosing the highest results. This cannot sufficiently support the conclusion that the method is consistently superior across different random seeds. In Table 1, none of the tasks reaches the conventional two-sided significance threshold of p < 0.05. The current sample size is not sufficient to confirm these improvements on its own. Of course, the authors also acknowledge this in lines 369 to 375. I would only consider increasing my score after the authors disclose the selection process for the highest GEARS result on each task. After the selection is complete, they should also reevaluate the method on an independent, pre-specified formal test set.
2.The compared baselines are relatively old. Recent methods are missing, and the representative baseline DP3 is also missing.
3.The real-robot experiments involve relatively simple tasks and are limited in scale.
4.The manuscript is not clearly written. Sections 3.2 and 3.3 are difficult to understand, and the motivation is not clearly explained. See the Questions section below for details.

Questions for authors

1.In Figure 2, are (3), the 6-DoF dexterous hand, and (4), the 7-DoF robotic arm, drawn in reverse?
2.Should the content in lines 175 to 178 be placed in Section 3.3? When readers look at the first part of Figure 1, the sudden mention of the student may confuse them.
3.In line 205, what is S in the MDP? Has it been mentioned? I may have missed it. If so, I hope the authors can point it out to me. If it has not been clearly explained, the authors could improve the wording here. Also, what does "γ is the discount factor" mean here? How should this be understood? I feel that this part is a little difficult for readers to understand. I suggest that the authors explain it clearly in natural language in the manuscript. The authors should ensure that every symbol here is clearly defined.
4.In lines 206 and 209, the sentence "The privileged expert observes ξ together with ..." is very difficult to understand. What does it mean? Is the sentence structure correct? For "joint positions and velocities, object poses and velocities, fingertip velocities, relative fingertip-object positions, contact state, and the previous action", are these items in a parallel relationship, or is there some other relationship? The repeated use of "and" here makes me very confused. The sentence seems to mean that ξ is listed alongside object poses, velocities, and other quantities. However, line 209 says that ξ contains object poses. This is contradictory. The authors do not define many symbols here, but the explanation seems confusing. This makes it difficult for me to understand. I also question whether these symbols really need to be defined. Is it really impossible to explain the same content clearly in natural language? If my understanding is incorrect, I hope the authors can correct me. I suggest that if the authors want to define equations and symbols, they should explain every symbol clearly. They should not leave readers and reviewers to guess what these symbols mean. Equations 1 and 2 are written clearly, but line 205 becomes very confusing.
5.In lines 222 to 223, what does "ten adaptive ranges" mean? I do not recommend referring to the appendix in this way. A reference like the one in line 215 is acceptable, but the description here is difficult to understand. The main text should allow readers to understand the content without having to read the appendix.
6.Does the manuscript explain why the "Automatic Domain Randomization via VP-SVGD" component is used? What problems does it solve?
7.Line 369 states, "The GEARS row contains the highest recorded per task rates for the full system." Does this mean that the comparison is unfair?
8.The No ADR curve in Figure 3(d) does not seem reasonable. It remains almost zero before 4 × 10^8 environment steps. Is there a problem with the experiment? Also, this contradicts the statement in line 429: "In Fig. 3(d), the VP-SVGD run reaches its final success level with fewer environment steps than the uniform-ADR run." Uniform ADR clearly reaches a stable level earlier than the method proposed by the authors.
9.Line 381 states, "Appendix C.1 states the remaining ambiguity in the recorded protocol for this control." Where is the relevant content in Appendix C.1? When using AI-assisted writing, the authors must carefully check the content.
10.Similarly, line 431 states, "Appendix B.3 reports the available cost information." I could not find the relevant content in the appendix either. This may be caused by an AI hallucination.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 8m8E (1/5)

Q1. Were the reported results obtained by selecting the best seeds or evaluation batches? The task-level comparisons are not significant at p<0.05p < 0.05. Please disclose the selection process and reevaluate the selected models on an independent, prespecified test set. (W1,Q7)

A1: Each method and task in the original Table 1 had three training runs, with fixed encoder checkpoints and seeds affecting teacher and student training. We evaluated one student checkpoint per run in 20 real-world trials and reported the highest of the three success rates. GEARS and every baseline followed the same procedure, with no selection among multiple checkpoints. The new independent evaluation includes all three policies, with deployment settings, initial-condition sampling procedures, and success criteria fixed before testing and no post-test selection or tuning. Each policy completes 20 trials per task, and all three seeds contribute equally to the mean. GEARS reaches 67.967.9\\%, outperforming π0.5\pi_{0.5} at 54.654.6\\%, ACT + Geo at 43.343.3\\%, DP + Theia at 42.142.1\\%, DP + SPA at 40.440.4\\%, and DP3 at 25.825.8\\%. GEARS also outperforms every baseline on each task (Table 1 below), so its advantage persists when all three seeds are included in independent testing.

We will replace the selected maxima with three-run means and present the original and independent tests separately in the revised manuscript. The appendix of the revised manuscript will provide per-seed success counts and standard deviations, together with task-level confidence intervals, to show variation across runs and uncertainty in the measured success rates.

(Table 1 appears in Part 2/5; references are provided at the end of Part 5/5.)

AuthorsOpenReview ↗

Response to Reviewer 8m8E (2/5)

(Continued from Part 1/5.)

Q2. The baselines omit recent methods and the representative DP3 baseline. How does GEARS compare with these alternatives? (W2)

A2: Experimental setup: We extend the comparison to a pretrained VLA, two robot-specific encoders, and an RGB-based DP3 adaptation. For each training seed and task, all six methods use the same 400 successful simulated expert trajectories. They receive observations from the same two RGB cameras and robot proprioception. All methods predict the same 26 robot commands: 14 arm-joint velocity targets and 12 hand-joint position targets. We fine-tune π0.5\pi_{0.5} [1] with a fixed instruction per task and pair frozen SPA and Theia encoders [2-3] with the same diffusion architecture as GEARS. DP3 [4] receives point clouds reconstructed from depth predicted by a separate, fixed metric-depth model based on Depth Anything V2 [5]. The same depth-prediction and calibrated back-projection pipeline processes simulated and real images. Task-specific policy training uses only simulated demonstrations. Each method is evaluated with three training seeds and 20 real-world trials per task and seed. Checkpoints, deployment settings, initial-condition sampling procedures, and success criteria are fixed before testing. The revised appendix will specify the pretrained checkpoints, trainable modules, training budgets, and input/output adaptations, including the metric-depth model and point-cloud preprocessing used for DP3.

Results and analysis: GEARS outperforms every baseline on all four tasks, with 67.967.9\\% mean success versus 54.654.6\\% for the strongest baseline, π0.5\pi_{0.5}, and 25.825.8\\% for DP3 with RGB-predicted depth. With the demonstrations and diffusion architecture fixed, replacing Geo with SPA or Theia reduces mean success by 40.540.5\\% and 38.038.0\\%, respectively, relative to GEARS. In a matched three-task ablation with the same Depth Anything V2 initialization, geometry fine-tuning raises success from 32.232.2\\% to 70.670.6\\%, more than doubling the success rate. These results demonstrate that task-specific geometric learning improves visual transfer and that GEARS achieves higher real-world success than the pretrained VLA and policies using alternative visual representations under the same demonstration budget.

Table 1. Real-world success rates (\\%). Three seeds ×\times 20 trials per task; 60 trials per task and 240 per method. Avg. is the equally weighted mean over four tasks, computed before rounding.

MethodGraspingPouringHandoverLiftingAvg.
ACT + Geo (retest)71.741.723.336.743.3
DP + SPA46.741.738.335.040.4
DP + Theia51.748.335.033.342.1
DP3 (RGB-pred. depth)36.731.718.316.725.8
π0.5\pi_{0.5} (sim fine-tuning)76.750.043.348.354.6
GEARS88.366.756.760.067.9

Note: ACT + Geo and GEARS are independently reevaluated. DP3 uses RGB-predicted metric depth. Relative decreases use GEARS as the denominator and are computed before rounding.

(References are provided at the end of Part 5/5.)

AuthorsOpenReview ↗

Response to Reviewer 8m8E (3/5)

(Continued from Part 2/5.)

Q3. The real-robot tasks are relatively simple and the evaluation is limited in scale. What conclusions does this evaluation support? (W3)

A3: Our study addresses three challenges in visual sim-to-real transfer for dexterous manipulation:

  1. Visual perception: The student must infer contact-relevant geometry from RGB despite appearance changes between simulation and reality. This is particularly challenging for transparent objects, whose appearance depends strongly on background and lighting.
  2. Policy learning: The RL teacher jointly controls the arms and hands using 14 arm-joint velocity commands and 12 hand-joint position commands across varying physical conditions. The student learns to perform these tasks from RGB and proprioception, while the teacher also observes object state, contact state, and physical parameters.
  3. Task execution: The difficulty lies in maintaining stable contact while coordinating orientation and timing across the arms and hands. Pouring requires regulating the amount of liquid transferred without losing grasp stability. Handover requires the receiving hand to secure the object before release, while bimanual lifting requires synchronized motion to keep the container level. Small errors in contact timing, vessel orientation, or relative hand motion can cause drops or spills.

The new comparison covers six methods, four tasks, three training seeds, and 20 trials per task and seed: 1,440 real-world executions. GEARS achieves the highest success rate on all four tasks, and the component experiments identify geometric learning and adaptive expert training as contributors to this advantage. The evaluation therefore demonstrates improved RGB-based sim-to-real transfer across distinct grasping, pouring, and bimanual coordination requirements. We will make these challenges and the task-specific success criteria explicit in the revised manuscript.

Q4. Why does No ADR remain near zero until approximately 4×1084\times10^8 steps? Fig. 3(d) also appears inconsistent with the claim that VP-SVGD reaches its final success level earlier than uniform ADR. (Q8)

A4: Training behavior: No ADR retains physical and visual randomization within fixed ranges. The initial policy rarely succeeds under difficult conditions, so learning starts slowly. VP-SVGD adjusts the ranges to the teacher's progress: low success triggers contraction, intermediate success keeps the ranges fixed, and higher success permits particle-based updates. In Fig. 3(d), Uniform ADR reaches a lower plateau earlier, while VP-SVGD continues to improve and finishes at a higher success rate. We will replace the statement at line 429 with this distinction and define fixed-range DR consistently in the text, tables, and figure caption.

Experimental setup: We evaluate checkpoints from the three-seed SAC runs that provide the demonstrations for the students in Table 4. Evaluation covers Grasping, Pouring, and Handover, using the same fixed distribution of initial conditions and physical parameters for every method. Each checkpoint is tested for 100 episodes per task and seed. Training budgets include all simulator interactions, including Active DR's reference rollouts [6].

Results and analysis: Uniform ADR leads at 10810^8 interactions with 49.049.0\\% success. VP-SVGD overtakes it by 3×1083\times10^8 and continues from 78.078.0\\% to 89.089.0\\% over the remaining budget. Active DR and Uniform ADR improve from 71.071.0\\% to 74.074.0\\% and from 63.063.0\\% to 66.066.0\\%, respectively; fixed-range DR finishes at 48.048.0\\%. Thus, VP-SVGD achieves the highest final success under the same interaction budget. Its advantage is higher performance with continued training, while Uniform ADR learns faster initially.

Table 2. Simulation success at matched training budgets (\\%). Grasping, Pouring, and Handover; three training seeds per method and task; 100 evaluation episodes per task, seed, and checkpoint. Entries average equally over tasks and seeds.

Method1×1081\times10^83×1083\times10^85×1085\times10^8
GEARS / VP-SVGD36.078.089.0
Uniform ADR49.063.066.0
Active DR39.071.074.0
No ADR (fixed-range DR)2.08.048.0

(Table 4 appears in Part 5/5; references are provided at the end of Part 5/5.)

Reviewer_ccv4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose zero-shot sim-to-real framework. A visual foundation model is fine-tuned with privileged inverse-depth and surface-normal supervision. A privileged expert is trained with VP-SVGD which uses an estimated failure landscape to adapt the object-pose and physical randomization bounds. The main contribution is a sim-to-real distillation design that connects what the student can perceive with how it represents expert actions.

Critical strengths

S1. The paper includes numerous ablation experiments and supplementary materials.

Critical weaknesses

W1. The methodological novelty needs clearer positioning. The framework builds on established geometric encoders and diffusion policies. Active Domain Randomization already uses learned signals and SVPG particles to select informative environments. GEARS introduces failure-based virtual-particle bound adaptation, but comparison only against uniform ADR does not establish its advantage over related adaptive methods.

W2. Evaluation selection and training variability remain unclear. Section 4.2 reports the “highest recorded per-task rates” without clearly specifying the selection procedure or equivalent treatment of baselines. Furthermore, each ablation uses one trained policy, leaving the repeatability of the component gains and encoder-policy interaction unresolved.

Questions for authors

Q1. Table 2a supports joint depth and normal supervision. What advantage does a separately supervised normal head provide over deriving normals directly from predicted depth?

Q2. How were the highest per-task success rates selected, how many candidates were considered, and did all baselines undergo the same selection procedure?

Q3. What distinguishes VP-SVGD technically and empirically from Active Domain Randomization using SVPG?

Q4. Does Appendix A.1 compare action distributions conditioned on the same student observation? What evidence supports attributing the observed action variation to hidden simulator states and physical parameters?

Minor presentation comment: In Figure 2, the hand and arm labels numbered 3 and 4 appear reversed relative to the legend. Please check their consistency.

5 comments
AuthorsOpenReview ↗

Response to Reviewer ccv4 (1/3)

Q1. How were the highest per-task success rates selected, how many candidates were considered, and did the baselines undergo the same selection procedure? (W2,Q2)

A1: Each method and task in manuscript Table 1 had three training runs. We evaluated one student checkpoint per run in 20 real-world trials and reported the highest of the three success rates. GEARS and all baselines followed the same procedure, with no selection among multiple checkpoints. Encoder checkpoints were fixed; the seeds affected teacher and student training. In the revised manuscript, we will replace the selected maxima with means over all three runs. Per-seed success counts and standard deviations will be provided in its appendix.

The new independent tests evaluate the policies from all three seeds with checkpoints, deployment settings, initial-condition sampling procedures, and success criteria fixed in advance. Every reported mean includes all three seeds, with no post-test selection or tuning. We will present these independent results separately from the original trials.

Q2. Each ablation uses one trained policy. Are the component gains and the encoder–policy interaction repeatable across training seeds? (W2)

A2: We evaluate component gains and the encoder–policy interaction across three teacher and student training seeds on Grasping, Pouring, and Handover. Each student completes 20 real-world trials per task under the same initial-condition sampling procedures and success criteria. Each encoder variant uses one fixed checkpoint across seeds. Tables 1–3 report success rates averaged over all three seeds: 60 trials per task and 180 per configuration. Avg. gives equal weight to each task and is computed before rounding.

Experiment 1: Geometric encoder. The methods share the expert demonstrations within each seed and use the same diffusion student architecture. The ablations remove normal supervision, inverse-depth supervision, depth–normal consistency, or all task-specific geometry fine-tuning. In the last setting, the DINOv2 backbone and DPT neck retain their Depth Anything V2 Small weights [1] and remain frozen while the student is trained. Geometry fine-tuning raises mean success from 32.232.2\\% to 70.670.6\\%. Removing either supervised target lowers success on every task, with a larger loss from removing inverse-depth supervision. Adding the consistency loss to joint depth and normal supervision further raises success from 62.862.8\\% to 70.670.6\\%. The full encoder performs best on every task: both depth and normal supervision improve transfer, and enforcing agreement between the two predictions provides an additional gain.

Table 1. Geometric encoder ablations.

ConfigurationGraspingPouringHandoverAvg. (\\%)
GEARS88.366.756.770.6
w/o normal supervision68.351.745.055.0
w/o depth supervision53.343.331.742.8
w/o depth–normal consistency75.060.053.362.8
w/o geometry fine-tuning41.733.321.732.2

Experiment 2: Domain randomization. All methods use SAC and 5×1085\times10^8 total simulator interactions per task and seed, including Active DR's reference-environment interactions [2]. Each teacher supplies 400 successful demonstrations per task to a student with the same Geo encoder and diffusion architecture. No ADR keeps physical and visual randomization within fixed ranges. Uniform ADR adapts the ranges using success thresholds; VP-SVGD additionally uses a learned failure model to determine how each parameter range changes. VP-SVGD achieves 70.670.6\\% mean success, compared with 58.958.9\\% for Active DR, 55.655.6\\% for Uniform ADR, and 36.136.1\\% for fixed-range DR. It outperforms all three alternatives on every task. With the student architecture, encoder, and demonstration count fixed, training the expert with VP-SVGD yields better real-world policies under the same interaction budget.

Table 2. Domain randomization comparison.

RandomizationGraspingPouringHandoverAvg. (\\%)
GEARS / VP-SVGD88.366.756.770.6
Uniform ADR70.048.348.355.6
Active DR71.753.351.758.9
No ADR (fixed-range DR)50.031.726.736.1

(References are provided at the end of Part 3/3.)

AuthorsOpenReview ↗

Response to Reviewer ccv4 (2/3)

(Continued from Part 1/3.)

A2 (continued):

Experiment 3: Encoder–policy interaction. We train BC and diffusion policy (DP) with the same expert demonstrations within each seed. DINOv2 uses the original self-supervised weights; Geo uses Depth Anything V2 initialization followed by task-specific geometry fine-tuning. Table 1 separately isolates fine-tuning with a shared initialization. Replacing DINOv2 with Geo raises mean success from 9.49.4\\% to 25.025.0\\% for BC and from 25.025.0\\% to 70.670.6\\% for diffusion. Geo improves both policies on every task, with larger gains for diffusion. Pairing Geo with diffusion delivers the highest success across all three tasks, reaching more than twice the mean success of either BC + Geo or diffusion + DINOv2.

Table 3. Encoder–policy comparison.

Policy + encoderGraspingPouringHandoverAvg. (\\%)
BC + DINOv220.08.30.09.4
DP + DINOv243.321.710.025.0
BC + Geo50.016.78.325.0
DP + Geo88.366.756.770.6

The three-run averages confirm gains from the geometric encoder and VP-SVGD on every task, together with a larger benefit from Geo when paired with diffusion. We will provide per-seed success counts for all configurations in Tables 1–3 in the appendix of the revised manuscript.

Q3. What distinguishes VP-SVGD from Active Domain Randomization with SVPG, and what evidence supports the distinction? (W1,Q3)

A3: Technical distinction. Active DR uses SVPG to train policies that propose environment parameters, with a discriminator reward derived from randomized and reference rollouts [2]. VP-SVGD trains a failure model over environment parameters using the teacher's task outcomes as labels, then applies SVGD directly to parameter vectors. These virtual particles set the sampling bounds, within which the simulator samples uniformly. Failure gradients guide the particles toward difficult conditions, kernel repulsion spreads them, and particle percentiles determine the new bounds for each parameter. The teacher's success rate determines whether the ranges contract, remain fixed, or receive this update. VP-SVGD therefore adapts the training range directly from task performance using existing teacher rollouts, without requiring reference-environment rollouts or a separately trained sampling policy.

Experimental evidence. At 5×1085\times10^8 total simulator interactions per task and seed, VP-SVGD achieves 70.670.6\\% mean real-world success versus 58.958.9\\% for Active DR (Table 2), a 19.819.8\\% relative improvement, with higher success on all three tasks. Both methods use the same SAC learner and supply 400 successful demonstrations per task to students with the same Geo encoder and diffusion architecture. VP-SVGD thus delivers better real-world transfer for the same amount of simulator interaction, with the gain coming from how the expert is trained.

Q4. What does a separately supervised normal head provide beyond computing normals directly from predicted depth? (Q1)

A4: Rationale. A separate normal head gives the encoder a direct training signal for surface orientation. This signal does not depend entirely on depth accuracy: the head predicts normals from shared features, whereas depth-derived normals depend on local depth differences that can amplify prediction errors. The depth–normal consistency loss then penalizes disagreement between the two predictions, encouraging them to describe the same local surface.

(Tables 1–2 appear in Part 1/3; references are provided at the end of Part 3/3.)

AuthorsOpenReview ↗

Response to Reviewer ccv4 (3/3)

(Continued from Part 2/3.)

A4 (continued):

Controlled comparison. The derived-normal and separate-head variants in Table 4 both receive normal supervision and smoothness losses, and both omit consistency. The separate head reduces mean angular error from 2.8∘2.8^\circ to 2.1∘2.1^\circ and depth Abs Rel from 0.043 to 0.031, while increasing real-world success from 56.356.3\\% to 62.962.9\\%. Since the prediction heads are removed at deployment, this success gain comes through the shared encoder features used by the policy. Adding consistency improves all three measures further, to 1.1∘1.1^\circ, 0.023, and 67.967.9\\%, respectively. The separate head and consistency loss therefore improve both geometric accuracy and downstream manipulation. We will add this comparison to Appendix C.1.

Table 4. Ablation of the normal prediction head. These are new measurements on the same 2,500-image simulation validation set used in manuscript Table 8, with the same median depth scaling and validity masks. Real SR averages Grasping, Pouring, Handover, and Lifting: three seeds ×\times 20 trials per task, 240 trials per method.

Geometry trainingDepth Abs Rel ↓\downarrowDepth δ1\delta_1 (\\%) ↑\uparrowNormal mean (∘^\circ) ↓\downarrowNormal $ < 11.25^\circ$ (\\%) ↑\uparrowReal SR (\\%) ↑\uparrow
Depth only0.04197.515.412.853.8
Derived-normal supervision0.04397.62.887.956.3
Separate heads, no consistency0.03198.32.196.662.9
Full Geo0.02399.81.199.567.9

Note: The metrics follow manuscript Table 8 and standard depth and normal evaluations [1],[3]. Abs Rel is mean absolute relative depth error; δ1\delta_1 is the percentage of valid pixels satisfying max⁡(d^/d∗,d∗/d^)<1.25\max(\hat d/d^*,d^*/\hat d) < 1.25 after median scaling. Normal mean is mean angular error; the next column reports the percentage below 11.25∘11.25^\circ. The first two rows use depth-derived normals, and the last two use the normal-head output.

Q5. Does Appendix A.1 compare action distributions conditioned on the same student observation? What evidence attributes the action variation to hidden simulator states and physical parameters? (Q4)

A5: Yes. Fig. 4 uses one fixed student observation: the same pair of RGB images and proprioceptive vector at the start of each chunk. Each expert point represents an action chunk generated under one of the physical-parameter settings. Across the three groups, the initial simulator state and teacher checkpoint are fixed; only the physical parameters change. The teacher observes these parameters, so it can generate different chunks for the same student input. BC and diffusion [4] receive that input without the physical parameters and predict chunks of the same length.

In the shared two-dimensional PCA projection, the BC prediction lies between regions occupied by expert chunks, while diffusion samples overlap several of those regions. Squared-error BC is trained to predict the conditional mean; diffusion models the conditional distribution and can represent several expert action patterns at the same input. The figure provides a qualitative comparison of this difference across the selected physical conditions. In the revised manuscript, Appendix A.1 and the caption will specify each group's parameter values and sampling rules, how the teacher generates each expert chunk, and the common prediction horizon.

Q6. Are the hand and arm labels numbered 3 and 4 reversed relative to the legend? (M1)

A6: Yes. Label 3 denotes the 7-DoF arms, and label 4 denotes the 6-DoF hands. We will correct the labels and caption in the revised version.

Reference:
[1] Depth Anything V2. NeurIPS, 2024.
[2] Active Domain Randomization. CoRL, 2019.
[3] Rethinking Inductive Biases for Surface Normal Estimation. CVPR, 2024.
[4] Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS, 2023.

Reviewer_eCgzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper is about sim-to-real dexterous manipulation, the policy is totally trained in simulation and zero-shot transfer to real robot. The author think the hardness comes from the visual gap and dynamic gap, the former means the rendered rgb is different from sim to real and the depth of transparent object is unreliable, and the latter means that in the teacher-student framework, the student network may learn the average of actions due to the lack of privileged info. So the paper propose a three-stages method:

  1. geometry aware perception: a visual encoder based on Depth Anything V2-Small, the author fine-tuned the depthanything v2 according to the rgb in simulation(label: depth normal). The encoder could be aware of the orientation and depth instead of outlook.
  2. VP-SVGD(Adaptive rl expert): a method of automatic domain randomization, using the fail distribution of in the range to decide where and how to expand.
  3. A diffusion-base distillation method Exp. The exp are tested on a self-designed chemical task bench, the performance of this work is the sota method among the method it compared. However, all five baselines in Table 1 are ablations of GEARS itself: each fixes one GEARS component and swaps the other. This shows that each component helps, but provides no system-level comparison against current methods. The paper excludes pretrained VLA policies because their data and embodiments differ, but this doesn’t hold: a pretrained policy such as pi0.5 and etc. can be fine-tuned with the simulation data. Whether large-scale pretrained visual and action priors transfer better than the proposed geometry-supervised encoder is precisely the question a reader would ask. Likewise, the robot-specific encoders discussed in Related Work are not sufficiently evaluated In short, this paper fine-tuned the DepthAnything v2 as a visual encoder, and propose a method of adaptive domain randomization which is useful in SAC framework, then a diffusion-base distillation is applied.
Critical strengths

This paper propose a three-stage framework to achieve available results of sim2real dexterous manipulation. The fine-tuned Depth Anything v2 is validated to be useful as a visual encoder to extract important visual info. And the method of adaptive rl expert is novel and useful. Then it use the diffusion model to distill the ability from teacher model to student model, proved to be effective. Also the videos of real dexterous manipulation in the sup. material are acceptable.

Critical weaknesses
  1. The method of diffusion distillation is existing: distilling a privileged, domain-randomized RL teacher into a diffusion student has been done in ClutterDexGrasp (2025), which trains a privileged teacher with a curriculum and distills it into a DP3 student for zero-shot sim-to-real dexterous grasping.
  2. The experiments lack baselines with strong pretrained VLA-style policies: the five baselines in Table 1 are likely the ablations of GEARS itself. The paper excludes pretrained VLA policies because their data and embodiments differ, but this doesn’t hold: a pretrained policy such as pi0.5 and etc. can be fine-tuned with the simulation data. Whether large-scale pretrained visual and action priors transfer better than the proposed geometry-supervised encoder is precisely the question a reader would ask.
Questions for authors
  1. Please fine-tune at least one pretrained VLA policy on on exactly the same simulated expert demonstrations and evaluate it zero-shot under the same real-world protocol. If feasible, also add a recent robot-specific encoder discussed in Related Work. This would show whether large-scale pretrained visual and action priors transfer better or worse than the proposed geometry-supervised encoder.
  2. Several references have incorrect or fabricated author lists. For example, “Cheng et al., 2025” (VIRAL, arXiv:2511.15200) lists none of the actual authors;Please verify and correct every reference.
2 comments
AuthorsOpenReview ↗

Response to Reviewer eCgz (1/2)

Q1. ClutterDexGrasp already distills a privileged RL teacher into a diffusion student for zero-shot sim-to-real dexterous grasping. What does GEARS contribute beyond this existing approach? (W1)

A1: GEARS builds on privileged teacher–student distillation [1-2], including the diffusion student in ClutterDexGrasp [1], and contributes task-specific geometric adaptation and failure-guided expert training. Appearance changes between simulation and reality make object geometry difficult to infer from RGB, especially for transparent objects. We fine-tune the encoder with simulated inverse depth and surface normals, using a consistency loss to enforce agreement between the corresponding predictions. The student therefore receives features trained to encode object shape and surface orientation without requiring depth measurements at deployment. Expert training faces a different problem: fixed randomization ranges can be too difficult early in training or too easy later. VP-SVGD estimates which parameter settings cause failures and uses virtual particles, each a vector of simulator parameters, to update the training ranges as the teacher improves. This makes range adaptation sensitive to where failures occur, while retaining uniform sampling within the updated bounds. Together, these contributions enable GEARS to achieve higher real-world success from the same expert-training interaction budget.

Both components improve real-world performance in the three-seed experiments on Grasping, Pouring, and Handover. Geometry fine-tuning raises mean success from 32.232.2\\% to 70.670.6\\% with the same Depth Anything V2 initialization. At the same teacher-training interaction budget, VP-SVGD reaches 70.670.6\\%, outperforming Uniform ADR at 55.655.6\\% and Active DR at 58.958.9\\% [8]. Each result averages 20 trials per task and seed. We will revise the contribution statement to distinguish these contributions from prior diffusion distillation.

Q2. The five baselines in the original Table 1 mainly isolate GEARS components. Please fine-tune a pretrained VLA on exactly the same simulated expert demonstrations and evaluate it under the same real-world protocol. If feasible, also evaluate a recent robot-specific encoder. (W2,Q1)

A2: Experimental setup: We extend the comparison to a pretrained VLA, two robot-specific encoders, and an RGB-based DP3 adaptation. For each training seed and task, all six methods use the same 400 successful simulated expert trajectories. They receive observations from the same two RGB cameras and robot proprioception. All methods predict the same 26 robot commands: 14 arm-joint velocity targets and 12 hand-joint position targets. We fine-tune π0.5\pi_{0.5} [3] with a fixed instruction per task and pair frozen SPA and Theia encoders [4-5] with the same diffusion architecture as GEARS. DP3 [6] receives point clouds reconstructed from depth predicted by a separate, fixed metric-depth model based on Depth Anything V2 [7]. The same depth-prediction and calibrated back-projection pipeline processes simulated and real images. Task-specific policy training uses only simulated demonstrations. Each method is evaluated with three training seeds and 20 real-world trials per task and seed. Checkpoints, deployment settings, initial-condition sampling procedures, and success criteria are fixed before testing. The revised appendix will specify the pretrained checkpoints, trainable modules, training budgets, and input/output adaptations, including the metric-depth model and point-cloud preprocessing used for DP3.

(References are provided at the end of Part 2/2.)

AuthorsOpenReview ↗

Response to Reviewer eCgz (2/2)

(Continued from Part 1/2.)

A2 (continued):

Results and analysis: GEARS outperforms every baseline on all four tasks, with 67.967.9\\% mean success versus 54.654.6\\% for the strongest baseline, π0.5\pi_{0.5}, and 25.825.8\\% for DP3 with RGB-predicted depth. With the demonstrations and diffusion architecture fixed, replacing Geo with SPA or Theia reduces mean success by 40.540.5\\% and 38.038.0\\%, respectively, relative to GEARS. The matched ablation in A1 further shows that geometry fine-tuning more than doubles success with the same pretrained initialization. These results demonstrate that task-specific geometric learning improves visual transfer and that GEARS achieves higher real-world success than the pretrained VLA and policies using alternative visual representations under the same demonstration budget.

Table 1. Real-world success rates (\\%). Three seeds ×\times 20 trials per task; 60 trials per task and 240 per method. Avg. is the equally weighted mean over four tasks, computed before rounding.

MethodGraspingPouringHandoverLiftingAvg.
ACT + Geo (retest)71.741.723.336.743.3
DP + SPA46.741.738.335.040.4
DP + Theia51.748.335.033.342.1
DP3 (RGB-pred. depth)36.731.718.316.725.8
π0.5\pi_{0.5} (sim fine-tuning)76.750.043.348.354.6
GEARS88.366.756.760.067.9

Note: ACT + Geo and GEARS are independently reevaluated. DP3 uses RGB-predicted metric depth. Relative decreases use GEARS as the denominator and are computed before rounding.

Q3. Manuscript revisions. (Q2)

A3: We will correct the author lists and bibliographic details, including the attribution of VIRAL to He et al., and verify every manuscript reference against its original publication. The Introduction and Related Work will credit prior diffusion distillation and state the geometric-adaptation and VP-SVGD contributions. We will include the VLA and encoder comparisons in A2 and document their pretrained checkpoints, trainable modules, training budgets, and input/output adaptations. Before each stage's equations, we will explain its purpose, inputs, training targets, and role at deployment. We will also check figure labels and appendix references against the revised content.

Reference:
[1] ClutterDexGrasp: A Sim-to-Real System for General Dexterous Grasping in Cluttered Scenes. CoRL, 2025.
[2] DextrAH-RGB: Visuomotor Policies to Grasp Anything with Dexterous Hands. ICRA, 2026.
[3] π0.5\pi_{0.5}: a Vision-Language-Action Model with Open-World Generalization. CoRL, 2025.
[4] SPA: 3D Spatial-Awareness Enables Effective Embodied Representation. ICLR, 2025.
[5] Theia: Distilling Diverse Vision Foundation Models for Robot Learning. CoRL, 2024.
[6] 3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations. RSS, 2024.
[7] Depth Anything V2. NeurIPS, 2024.
[8] Active Domain Randomization. CoRL, 2019.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.