acceptodds
Under review as a conference paper at ICLR 2027

Latent RL Priors for Data-Efficient Dexterous Manipulation

Abstract

Learning dexterous manipulation from limited visual demonstrations is challenging because policies must acquire both perceptual grounding and precise control, while reinforcement learning (RL) in simulation can acquire rich control behaviors from large-scale interaction with privileged information. We introduce Latent RL Prior, a two-stage framework that reuses this control knowledge for data-efficient visual policy learning. During privileged pretraining, we distill large-scale RL rollouts into a flow-matching action decoder conditioned on proprioception and latent task context. During visual adaptation, we freeze the decoder and learn a visual interface that maps images to its latent context using action supervision alone, without privileged labels or latent targets. The same decoder can further be shared across arms by learning coordinated visual contexts for bimanual manipulation. Across three simulated tasks, our method outperforms full-parameter fine-tuning of and GR00T N1.7 in the low-data regime; on door lifting and bimanual assembly, it surpasses full-data baselines using and fewer demonstrations, respectively. Further analysis shows that direct visual-to-latent conditioning improves adaptation performance and training stability over routing visual predictions through the pretrained state encoder. Robustness to unseen mass variation and greater linear accessibility of contact-force information in decoder representations further provide evidence of retained RL-acquired control knowledge. Training and evaluation code and datasets will be released.

Primary area:
applications to robotics, autonomy, planning
Keywords:
dexterous manipulation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_GctROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Learning dexterous manipulation from limited visual demonstrations is a major challenge, since policies must not only understand perceptual information but also possess fine-grained control. Reinforcement learning (RL) in simulation, by contrast, can leverage privileged information and accumulate rich control behaviors through large-scale interaction. Motivated by this, the authors propose Latent RL Prior, a two-stage framework designed to turn simulation-acquired control knowledge into data-efficient visual policies. The first stage, privileged pretraining, distills large-scale RL rollouts into a flow-matching action decoder conditioned on proprioception and latent task context. The second stage, visual adaptation, keeps the decoder frozen and trains only a visual interface via action supervision, mapping images to the corresponding latent context without privileged labels or latent targets. Moreover, the same decoder can be shared across arms by learning coordinated visual contexts for bimanual manipulation.

Critical strengths
  1. Latent RL prior achieves better performance than the baselines with less data.
  2. The method is simple and effective, and the paper is clearly written and easy to understand.
  3. The ablation studies effectively demonstrate the benefit of the prior.
Critical weaknesses
  1. The additional computational resources and training time introduced by the Latent RL prior are not reported.
  2. There is no validation through real-robot experiments.
  3. There is no discussion of whether the method offers generalization, for example, in coping with environmental changes.
Questions for authors
  1. I think this is a good idea, but it needs to be validated on a widely recognized benchmark or through real-robot experiments.
  2. How much additional computational resource and training time does the introduction of the Latent RL prior incur?
  3. Does the method provide a certain degree of generalization?
  4. Are the three tasks handled by a single model or by one model per task? Is the RL prior shared across the three tasks?
  5. What kind of network architecture is the Q-Former?
2 comments

Thank you for your time and for recognizing the simplicity of the method and the clarity of the paper. We appreciate your observation that the policy performed well with fewer visual demonstrations and that the ablations showed the contribution of the RL prior.

W1 / Q2. “additional computational resource and training time”

We reused the released SimToolReal teacher, whose authors report approximately 12 days on one A6000 GPU (288 GPU-hours) for the final Isaac Gym training run (General Response 3). We fine-tuned the checkpoint in MuJoCo Warp for 500 epochs with 18,432 parallel environments. This took 2.5 hours on one RTX PRO 6000 and raised the teacher’s success rate from 57% to 80.4%.

We then collected 100,000 successful simulated state–action rollouts and trained one action decoder. All four downstream tasks share this frozen decoder, so we incurred the distillation cost once.

W2 / Q1. “a widely recognized benchmark or through real-robot experiments”

We agree that evaluation on one simulated platform limits our conclusions. The SimToolReal teacher commands a KUKA iiwa in joint space, and our physical robot uses a different arm. We expect the main transfer challenge to be matching the arm and hand dynamics during contact. We therefore plan to calibrate the simulator to our hardware, retrain the RL teacher, and distill a new decoder. With the decoder frozen, we would train the visual interface on real camera images using only the teleoperator’s actions as supervision (General Response 4). Hardware validation and evaluation on an independent benchmark remain future work.

We added Grasp, a fourth simulated task that reuses the same frozen decoder as the original three tasks. The robot must lift a tabletop object at least 10 cm and hold it steady for 1.5 s. We used 96 training objects and 24 held-out objects from Google Scanned Objects, YCB, and Objaverse-LVIS. Our policy achieved higher grasp success than both baselines on training and held-out objects at all four demonstration budgets.

Grasp success (%) on training / held-out objects. Ours reports mean ± SD over three training seeds. Each baseline uses one seed.

EpisodesOursπ₀.₅GR00T N1.7
5047.9 ± 2.9 / 47.8 ± 2.830.7 / 33.732.4 / 31.2
10060.5 ± 1.9 / 63.6 ± 2.842.5 / 46.416.5 / 18.9
1,00082.1 ± 0.9 / 82.2 ± 2.373.7 / 72.966.4 / 63.2
4,000 (full)85.1 ± 0.5 / 84.4 ± 0.480.1 / 78.573.2 / 69.1

W3 / Q3. “Does the method provide a certain degree of generalization?”

Our policy generalizes to object masses unseen during visual adaptation (Fig. 8). Each task’s demonstrations used a single reference mass. We tested pose alignment and door lifting at higher masses without further training. Door-lifting success was similar at the reference mass. At 5.0 kg, our policy retained 32.6% success with 50 demonstrations, compared with 12.9% for π₀.₅ with 500. These results suggest that the visual policy inherits robustness to mass changes from the frozen RL prior, whose large-scale pretraining included randomized object densities.

Q4. “Is the RL prior shared across the three tasks?”

The four downstream tasks share one frozen action decoder. Each learns its own visual interface.

Q5. “What kind of network architecture is the Q-Former?”

The BLIP-2-style Q-Former has four pre-norm blocks (width 384, six heads, feed-forward width 1,024). In each block, learned queries attend to one another, cross-attend to DINOv2 ViT-S/14 patch tokens with learned camera embeddings, and pass through a feed-forward layer.

Door lifting uses six queries. Bimanual assembly uses one joint Q-Former with 12 queries, six per arm. Self-attention lets the two arms’ queries exchange information, and every query cross-attends to the front and both wrist views. We split the output into six context tokens per arm. The same frozen single-arm decoder predicts each arm’s action from its tokens and proprioception.

Thank you for your response. I think you have solved all my questions. I will raise my score to 3.

Reviewer_cDWEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Latent RL Prior, a 2-stage framework for transferring dexterous control skills learned through privileged reinforcement learning to visual manipulation policies. First, the author distill a privileged pretraining with 100k successful rollout into a flow-matching action decoder conditioned on proprioception history and latent task context. Duration visual adaptation, they freeze the decoder and train a visual interface using action supervision alone, without privileged state labels or latent targets.

The method is evaluated on three simulated tasks: targe pose alignment with an unseen brush, door lifting, and bimanual assembly. It outperforms the full-parameter fine-tuning of pi 0.5 and GR00T N1.7 baseline in a low-data regime. The paper claims it exceed full-data VLA baselines with 10x/5x fewer visual demonstration for each task. The paper also compares latent and explicit transfer, studies which model components to fine-tune, and test whether the adapted policy retains robustness to heavier objects and information about contact forces.

Critical strengths
  • This a practical transfer method for reusing privileged RL control skills. The visual interface is trained using action supervision alone, without privileged-state labels or paired RL trajectories. This makes the method suitable for demonstrations containing only images, proprioception, and actions. With the image encoder frozen, latent transfer consistently outperforms the same architecture trained from scratch, showed in Figure 5, supporting the contribution of the pretrained action decoder. Direct latent conditioning also achieves better performance and more stable training than the tested explicit-transfer interface.

  • The paper showed strong demonstration efficiency across different downstream tasks. Figure 4 shows strong results in the low-data regime against both evaluated VLA models, reported 10x and 5x visual demo reductions for door lifting and assembly. Also, for bimanual manipulation, both arms share the frozen decoder, while a joint visual Q-Former learns coordinated, arm-specific contexts, providing a simple way to reuse single-arm control skills in such tasks. The mains results runs on 3 runs, providing evidence of consistency across runs.

Critical weaknesses
  • Differences in training and model selection make the claimed demonstration savings difficult to assess. The proposed method is trained for 100k updates, while the VLA baseline receives 30k. Without baseline learning curves with longer training results, it is unclear whether the reported 10x or 5x saving would hold with further training happened. Each run also reports the best evaluation among the five checkpoints, it would be more convincing with showing converged baseline, together with checkpoint selection on a separate validation set would make the statement more solid.
Questions for authors
  • Were the VLA baselines evaluated with multiple learning rates, and were hyperparameters selected on a separate validation set? If so, do the claimed 10× and 5× demonstration savings hold under their best settings?
1 comment

Thank you for the time you devoted to reviewing our transfer experiments. We appreciate your observations that latent conditioning gave more stable training than the explicit interface and that our policy learned effectively from few visual demonstrations using action supervision alone.

W1. “Differences in training and model selection”

We thank the reviewer for raising this important concern regarding training convergence and the fairness of the comparison. We agree that ensuring convergence across different policy architectures and dataset sizes is challenging, particularly given their differences in model capacity, initialization, and optimization dynamics. In order to make sure the all policy are converged, we extended the training runs beyond our preliminary experiments and examined performance at multiple checkpoints. Specifically, our method was trained for up to 100k updates, while the VLA baselines, π₀.₅ and GR00T N1.7, were fine-tuned for up to 30k updates. All the results in the paper are trained in long training.

At each grasping checkpoint, we report success (%) on training / held-out objects. The columns give the number of training episodes. Our results are mean ± SD across three training seeds. Each baseline uses one seed.

Our method (50 and 100 episodes).

Updates50100
20k47.9 ± 2.9 / 47.8 ± 2.860.5 ± 1.9 / 63.6 ± 2.8
40k44.4 ± 3.7 / 44.3 ± 1.755.9 ± 1.4 / 59.1 ± 0.3
60k42.4 ± 2.4 / 43.9 ± 3.453.2 ± 1.0 / 56.9 ± 0.5
80k41.5 ± 2.3 / 43.5 ± 3.151.3 ± 1.9 / 54.6 ± 1.2
100k40.6 ± 3.3 / 42.7 ± 3.150.7 ± 2.1 / 53.6 ± 2.3

Our method (1,000 and 4,000 episodes).

Updates1,0004,000
20k81.7 ± 1.2 / 81.9 ± 2.583.8 ± 0.2 / 82.6 ± 0.5
40k81.6 ± 0.2 / 81.4 ± 1.284.5 ± 0.1 / 82.9 ± 1.9
60k81.0 ± 0.5 / 80.4 ± 0.885.0 ± 0.7 / 83.2 ± 1.1
80k80.6 ± 0.6 / 80.8 ± 0.184.1 ± 0.5 / 81.9 ± 1.7
100k80.4 ± 0.1 / 80.4 ± 1.283.6 ± 0.4 / 83.3 ± 1.5

π₀.₅ (all four dataset sizes).

Updates501001,0004,000
10k30.6 / 33.742.5 / 44.872.7 / 72.979.9 / 77.9
15k30.2 / 33.239.5 / 42.373.7 / 72.080.1 / 78.5
20k29.5 / 30.539.6 / 41.872.7 / 69.978.9 / 77.7
25k28.8 / 31.640.7 / 44.472.9 / 71.079.6 / 77.2
30k30.7 / 32.341.0 / 46.471.2 / 69.579.6 / 78.5

GR00T N1.7 (all four dataset sizes).

Updates501001,0004,000
10k23.7 / 23.42.6 / 1.661.1 / 57.040.1 / 35.8
15k32.4 / 31.27.3 / 7.363.5 / 62.165.0 / 60.9
20k28.3 / 28.46.4 / 6.666.4 / 63.265.2 / 66.0
25k30.6 / 27.914.5 / 15.665.5 / 62.172.4 / 69.0
30k29.1 / 29.616.5 / 18.966.1 / 60.473.2 / 69.1

These results suggest that the training budgets are sufficient for performance to stabilize in most evaluated settings. Our method reaches its highest or near-highest performance within the first 20k–60k updates, with no consistent improvement from further training. In the low-data regime, performance gradually decreases with additional updates, suggesting that longer training may lead to overfitting rather than improved generalization. VLA baselines are also converged for 30k training steps. gr00tn1.7 ep100 might be an excpetion due to bad training seed.

Q1. “Were the VLA baselines evaluated with multiple learning rates, and were hyperparameters selected on a separate validation set?” Regarding hyperparameter selection, we follow the officially recommended fine-tuning configurations for both π₀.₅ and GR00T N1.7, without conducting an additional learning-rate sweep or selecting hyperparameters on a separate validation set since it is too costly for full parameter finetuning.

We agree with the reviewer that the original claims of 10× and 5× demonstration savings may be too strong without such an exhaustive comparison. We will therefore revise our wording to emphasize that our approach achieves greater demonstration efficiency under the evaluated training configurations, particularly in the low-data regime.

Reviewer_ckojOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Latent RL Prior, a two-stage framework that reuses this control knowledge for data-efficient visual policy learning. During privileged pretraining, it distills large-scale RL rollouts into a flow-matching action decoder conditioned on proprioception and latent task context. During visual adaptation, the decoder is frozen and learns a visual interface that maps images to its latent context using action supervision alone, without privileged labels or latent targets.

Critical strengths
  1. The paper tackles a useful transfer problem. Preserving an RL-trained state-based controller while learning its visual interface is a practical approach to combining simulation experience with visual demonstrations.
  2. The downstream task results are appealing. The authors demonstrate that the proposed method performs better than other pretrained VLA models (π0.5\pi_{0.5}, GR00T). The single-arm-to-bimanual extension is also appealing and interesting. The assembly results suggest that this approach can support meaningful coordination. The paper also includes informative ablation results.
  3. The paper is well presented and illustrative. It's easy to follow.
Critical weaknesses
  1. More clarifications on novelty. The central design replaces the privileged-state encoder with a visual encoder while preserving a pretrained action decoder. This is intuitive and relevant, but closely related to established teacher–student and latent-adaptation approaches like RMA [1]. The paper should more clearly explain what these contributions enable beyond existing approaches and experimentally isolate their benefits.
  2. Experiment support for data-efficient claims. Evaluation covers only three simulated downstream tasks, with no physical-robot validation. Although the reported low-data improvements are encouraging, the full downstream datasets contain 4,000, 500, and 5,000 demonstrations, which are already a large number for specific tasks. Thus, the practical cost of obtaining downstream supervision remains substantial and insufficiently quantified. Also, the brush pose alignment task seems weird to me because original SimToolReal paper already demonstrates zero-shot generalization to unseen objects, including brushes not seen in training, so the very low success with ~200 demonstrations seems suspicious to me.
  3. Insufficient baselines. The paper only compares with π0.5\pi_{0.5} and GR00T, which combines difference in architecture, pretraining rollouts, finetuning strategies: only the proposed method gets access to the large-scale sim RL experiences, while baselines don't. Therefore, it's meaningful to at least mid-train the baselines with the same 100,000 rollouts. It would also be informative to compare against directly adapting the original SimToolReal controller through a learned visual interface, similar to Play2Perfect [2], and on more downstream tasks to demonstrate efficient task transfer.
  4. Some missing details. The paper does not adequately specify history and chunk lengths, executed chunk length, control frequency, action representation, decoder dimensions, token counts, inference integration steps, or complete optimization settings. The bimanual extension also needs details on left/right proprioceptive and action conventions. And the details for downstream task demonstrations acquisition are also unspecified. There is no supplementary materials and robot videos.

[1] Kumar, Ashish, et al. "Rma: Rapid motor adaptation for legged robots." arXiv preprint arXiv:2107.04034 (2021).

[2] Lum, Tyler Ga Wei, et al. "Play2Perfect: What Matters in Dexterous Play Pretraining for Precise Assembly?." arXiv preprint arXiv:2606.26428 (2026).

Questions for authors

Please see weaknesses

2 comments

Thank you for your careful review and for highlighting the practical value of combining RL experience with visual demonstrations. We appreciate your comments on the comparisons with pretrained VLAs, the bimanual assembly results, the ablation studies and paper writing.

W1. “More clarifications on novelty.”

We investigate where visual representations should interface with a pretrained RL controller when demonstrations do not provide privileged state labels. Existing approaches, such as RMA, train their adaptation modules with privileged state information supervision. However, obtaining such privileged information for visual demonstrations can be costly and impractical. We therefore investigate whether a pretrained RL controller can be effectively adapted using action supervision alone, and identify the key bottlenecks associated with this setting.

Specifically, we compare two interfaces to the same frozen action decoder, both trained using only action supervision. The explicit interface predicts the inputs to the pretrained state encoder, whereas the latent interface directly maps visual features into the action decoder's context space. Across all tested dataset sizes, the latent interface consistently achieves higher success rates with fewer training updates and exhibits greater stability across random seeds. To further isolate the effect of interface dimensionality, we introduce a bottleneck into the latent interface and find that reducing its dimensionality degrades performance. Nevertheless, even with a low-dimensional bottleneck, the latent interface continues to outperform the explicit interface while maintaining greater stability across training seeds. Additional results are provided in General Response #2.

These findings highlight that effectively leveraging a pretrained RL controller through action supervision alone is nontrivial, and that the choice of interface plays a critical role in enabling successful knowledge transfer. Our work not only proposes a simple yet effective latent interface but also provides a systematic analysis of its design considerations. Furthermore, we investigate what knowledge can be transferred from large-scale RL pretraining and provide insights into how and why such pretraining can enable data-efficient downstream visual policy learning.

W2. “Experiment support for data-efficient claims”

We acknowledge that the lack of physical-robot validation is a limitation of our work. Our paper primarily focuses on a fundamental question: how to effectively interface visual observations with a pretrained RL controller and leverage knowledge acquired through large-scale RL training to improve the data efficiency of downstream visual policy learning. We believe this is an important problem in both simulation and real-world settings.

We also acknowledge that datasets containing 500, 4,000, or 5,000 demonstrations represent substantial data budgets. However, we collect demonstrations at this scale primarily to systematically evaluate policy performance across different training dataset sizes, rather than because our approach requires such large amounts of data. Our results show that the benefits of our approach are particularly pronounced in the low-data regime. By extending our evaluation to larger datasets, we aim to investigate whether the advantages of an RL prior gradually diminish as the amount of visual demonstration data increases. We hypothesize that, given sufficiently large amounts of visual supervision, a policy may eventually learn the necessary dexterous manipulation skills directly from demonstrations, thereby reducing the relative benefit of RL pretraining.

Furthermore, it is important to distinguish the released SimToolReal RL checkpoint from an end-to-end visual policy. The SimToolReal checkpoint operates on state-based observations, requiring the current 6D object pose and the target pose as policy inputs. In its real-world deployment, these inputs are provided by separate components: a perception pipeline based on vision foundation models estimates the current object pose, while target poses are obtained from predefined task trajectories. In contrast, our setting considers visual policy learning without access to either the current object pose or the target pose as explicit inputs. The policy might instead need to learn to extract task-relevant information directly from visual observations using only action supervision, without explicit pose labels. SimToolReal uses multiple vision foundation models such as SAM and FoundationPose, which are trained with large scale labeled dataset. And in our case, we only have action as supervision.

W3. “Insufficient baselines.”

We agree that we need to add mid-train to VLA with 100,000 RL rollouts and see whether VLA can also benefit from the large scale RL training. It seems nontrivial and we cannot draw a solid conclusion during the rebuttal phase. We leave this for future research.

As for the baselines, we follow your suggestion and introduce an additional baseline, Teacher (Explicit), which projects visual observations into the explicit state space and directly connects them to the SimToolReal teacher checkpoint, further fine-tuned in our MuJoCo simulation. As the SimToolReal teacher uses an LSTM to model temporal dependencies, we employ teacher forcing to handle its recurrent hidden states during training. Both explicit-interface baselines share the same vision encoder architecture, and all three methods are trained with identical action supervision and learning-rate schedules to ensure a fair comparison.

As reported in the table below, our latent-interface approach consistently outperforms both explicit-interface baselines in mean success rate while requiring fewer training updates, across all three dataset sizes and both pose tolerances. In addition, it demonstrates lower variability across random seeds than the explicit-interface baselines in all six evaluation settings, further highlighting its advantages in training stability and data efficiency.

Brush alignment success (%) by training episodes (Eps.) and adaptation steps. Values are mean ± SD across five seeds, using each seed’s best evaluation within the stated budget. The explicit teacher is SimToolReal.

Eps.ModelSteps1.5 cm5 cm
200Latent (ours)50k1.7 ± 0.322.4 ± 1.3
200Explicit RL prior100k0.9 ± 0.614.0 ± 8.1
200Explicit teacher200k0.0 ± 0.01.0 ± 1.8
1,000Latent (ours)100k15.9 ± 0.650.7 ± 0.2
1,000Explicit RL prior200k4.4 ± 3.626.7 ± 12.5
1,000Explicit teacher200k0.1 ± 0.28.7 ± 7.1
4,000Latent (ours)200k36.9 ± 1.359.4 ± 1.1
4,000Explicit RL prior240k27.1 ± 9.952.7 ± 8.6
4,000Explicit teacher240k3.7 ± 5.224.7 ± 33.8

Besides, we would like to clarify an important distinction regarding SimToolReal and Play2Perfect. Both methods rely on pretrained vision foundation models, such as SAM and FoundationPose, to estimate explicit object poses, which are subsequently provided as inputs to their state-based RL controllers. In particular, Play2Perfect uses FoundationPose for 6D object pose tracking and derives target poses from the estimated fixture pose and predefined CAD transformations.

W4. “Some missing details.”

Thank you for pointing out these omissions. Videos demonstrating our model's performance across all four downstream tasks are provided in the supplementary materials. We will include these settings in the appendix and provide simulated rollout videos for all four tasks. Our policy uses only the current observation and predicts and executes one action per step. The VLAs use their official default action chunk lengths, and we execute only the first action in each chunk. We use the same 29-D action space as SimToolReal. The flow-matching action decoder (23.6M parameters) has eight transformer layers, a width of 384, six heads, and a feed-forward width of 1,024. It cross-attends to context and uses AdaLN-Zero conditioned on flow time and 109-D proprioception. Inference uses 10 Euler steps of size 0.1. The state Q-Former pools 10 tokens from the 31-D privileged state into two 384-D tokens. DINOv2 ViT-S/14 encodes each of three 224 × 224 views into 256 patch tokens. The visual Q-Former returns two 384-D tokens for brush alignment and grasping, six for door lifting, and six per arm for assembly. We use AdamW (learning rate and weight decay both 10−410^{-4}) with warmup over the first 0.01% of updates followed by cosine decay, a batch size of 256, gradient clipping at 1.0, an EMA decay of 0.999 for the weights, and FP16 mixed precision. DINOv2 uses 0.05× the base learning rate.

Reviewer_BosLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Latent RL Prior, a two-stage framework for data-efficient visual dexterous manipulation. In privileged pretraining, large-scale RL rollouts are distilled into a flow-matching action decoder conditioned on proprioception and latent task context. In visual adaptation, the decoder is frozen and a visual interface (DINOv2 + Q-Former) is learned to map images directly to the latent context using only action supervision, without privileged labels or latent targets. The same decoder is shared across arms for bimanual manipulation by learning coordinated visual contexts. Experiments on three simulated tasks (target pose alignment, door lifting, bimanual assembly) compare against full-parameter fine-tuning of π0.5 and GR00T N1.7, a from-scratch baseline, and an explicit-transfer variant. Additional analyses examine mass robustness and linear accessibility of contact-force information.

Critical strengths

1. The core idea is simple, principled, and well executed.
Freezing the RL-distilled action decoder and learning only a visual→latent interface with action supervision alone is an elegant way to reuse privileged RL control knowledge. It avoids the need for privileged labels, latent targets, or paired pretraining/adaptation trajectories, which is a meaningful practical advantage over RMA-style adaptation.

2. The bimanual extension is clean and non-trivial.
Sharing a single-arm frozen decoder across both arms and learning coordinated contexts through a joint Q-Former is a sensible design that reuses the single-arm prior while learning inter-arm coordination. This is a nice architectural contribution.

3. The analysis goes beyond success rates.
The mass-robustness test, linear contact-force probing, latent-vs-explicit transfer comparison, and learning-rate ablations provide useful insight into what the RL prior retains and how it should be adapted. The limitation section is honest about the scope of the claims.

Critical weaknesses

1. All experiments are in simulation, including visual demonstration collection.
The paper studies dexterous manipulation and claims data-efficient visual adaptation, but there is no real-robot validation. The sim-to-real gap for dexterous manipulation with visual policies is substantial, and the paper does not discuss whether the latent interface or the frozen decoder would transfer. The limitation section acknowledges this, but for a paper whose central claim is about visual policy learning, this is a significant gap.

2. The data-efficiency claim ignores the pretraining cost.
The method uses 100,000 successful RL rollouts for pretraining, while baselines use only the downstream demonstrations. The paper’s limitation section correctly notes that this is “downstream visual-demonstration efficiency, not total data or compute efficiency,” but the abstract and introduction state “10× and 5× fewer demonstrations” without this qualifier. The claim should be scoped explicitly in the abstract, or the paper should report total data/compute for a fairer comparison.

Questions for authors
  1. Can you report total pretraining data and compute (RL rollouts, GPU-hours) alongside downstream demonstration counts, and scope the “10×/5× fewer demonstrations” claim explicitly in the abstract?

  2. For bimanual assembly, how is inter-arm coordination represented in the joint Q-Former contexts? Did you compare against separate per-arm decoders or per-arm Q-Formers?

  3. What is the plan for real-robot validation, and what do you expect the main sim-to-real challenges to be for the frozen decoder and latent interface?

1 comment

Thank you for your time and for recognizing the value of adapting a frozen decoder with action supervision alone, without privileged labels. We appreciate your comments on bimanual coordination with a shared decoder and on our analysis of contact forces and robustness to changes in mass.

W2 / Q1. “The data-efficiency claim ignores the pretraining cost.”

We agree that the reported savings apply only to downstream visual demonstrations. The SimToolReal authors trained their teacher in Isaac Gym and report 288 GPU-hours for the final run. To adapt the released checkpoint to our MuJoCo simulator, we fine-tuned it for 2.5 hours on one RTX PRO 6000, raising the teacher’s success rate from 57% to 80.4%.

We propose revising the abstract and introduction to read, “We pretrained the action decoder on 100,000 successful simulated rollouts from a large-scale RL teacher. The resulting policy surpassed both VLA baselines using fewer downstream visual demonstrations.”

Q2. “For bimanual assembly, how is inter-arm coordination represented in the joint Q-Former contexts? Did you compare against separate per-arm decoders or per-arm Q-Formers?”

Each arm’s visual context can incorporate information from the other arm because all 12 learned queries (six per arm) interact through self-attention. In each block, the queries then use cross-attention to read features from the front and both wrist cameras. We call this design the joint Q-Former. We split its output into six context tokens per arm. The same frozen action decoder, pretrained on a single arm, predicts each arm’s action from its tokens and proprioception (Sec. 3.3).

Following your suggestion, we compared this design with two per-arm Q-Formers. Each has its own weights and six queries that cross-attend to all three views. Self-attention operates only within each arm. To test reuse of the single-arm prior, we kept the decoder shared in both variants and did not evaluate separate per-arm decoders.

The two designs achieved similar mean assembly success across all five dataset sizes, with a maximum difference of 1.5 percentage points in the table below. Our design supports cross-arm interaction with fewer trainable parameters.

Bimanual assembly success (%) with joint and separate Q-Formers. Values are mean ± SD across training seeds.

EpisodesJointSeparate
20029.9 ± 3.931.1 ± 7.9
50056.6 ± 2.956.2 ± 5.1
1,00072.3 ± 2.970.8 ± 1.3
2,00080.3 ± 3.880.1 ± 2.8
5,000 (full)84.2 ± 3.984.2 ± 3.8

W1 / Q3. “What is the plan for real-robot validation, and what do you expect the main sim-to-real challenges to be for the frozen decoder and latent interface?”

Real-robot validation plan. We will first calibrate the simulated arm and hand to match the dynamics of our physical system. SimToolReal’s policy commands a KUKA iiwa in joint space. Because our hardware uses a different arm, we plan to retrain the RL policy in the calibrated simulator and distill its rollouts into a new action decoder. We will then collect multi-camera images and corresponding actions through teleoperation to train the visual interface with the decoder frozen. We will compare task success on hardware with π₀.₅ and GR00T N1.7 fine-tuned on the same demonstrations.

Expected transfer challenges. We expect the main sim-to-real challenge to lie in the frozen RL prior. The same joint command may produce different hand motion or contact forces on hardware, and retraining the RL policy after calibration takes substantial time. Because the latent interface will learn directly from real images and actions, we expect visual domain shift to be less of a concern. Our simulation results show that the latent interface can learn from few visual demonstrations using action supervision alone. We expect this benefit to extend to hardware if the simulator accurately captures the arm and hand dynamics.

Our current experiments study what large-scale RL pretraining contributes to data-efficient dexterous manipulation under action supervision only, without privileged state labels or latent targets. On four simulation tasks, mapping vision directly to the decoder’s latent context yielded higher mean success and less variation across seeds than predicting the state encoder’s inputs. We agree that hardware validation is necessary and are working toward it.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response

We thank the reviewers for their feedback, suggestions, and recognition of our contributions. We list the common questions and response here.

1. Additional task validation

To further evaluate the performance of our approach, we design an additional task in which the robot must grasp an object from a tabletop and hold it in the air for 1.5 seconds. The object is randomly positioned on the tabletop. We select 120 objects from Google Scanned Objects, YCB, and Objaverse-LVIS, and construct a stratified split across eight grasp families, comprising 96 training objects and 24 held-out test objects. We collect 4,000 successful demonstrations using only the training objects and use them to train our method and fine-tune two baselines, π₀.₅ and GR00T N1.7. We train our method with three random training seeds, while fine-tuning each baseline with a single seed, as neither baseline introduces randomly initialized parameters. The results in the table below demonstrate that our approach consistently outperforms the baselines across varying training dataset sizes, on both seen and unseen objects.

Grasp success (%) on training / held-out objects. Ours reports mean ± SD over three training seeds. Each baseline uses one seed.

EpisodesOursπ₀.₅GR00T N1.7
5047.9 ± 2.9 / 47.8 ± 2.830.7 / 33.732.4 / 31.2
10060.5 ± 1.9 / 63.6 ± 2.842.5 / 46.416.5 / 18.9
1,00082.1 ± 0.9 / 82.2 ± 2.373.7 / 72.966.4 / 63.2
4,000 (full)85.1 ± 0.5 / 84.4 ± 0.480.1 / 78.573.2 / 69.1

2. More interface baselines

Our approach employs a latent interface to bridge the vision encoder and action decoder. We compare our method against two explicit-interface baselines: RL Prior (Explicit) and Teacher (Explicit). RL Prior (Explicit) maps visual observations into the explicit state space, whose dimensionality matches that of the privileged state information, and feeds the predicted states into the pretrained state-based policy. Similarly, Teacher (Explicit) maps visual observations into the explicit state space but directly uses the SimToolReal teacher checkpoint, which we further fine-tune in our MuJoCo simulation. Since the SimToolReal teacher employs an LSTM to capture temporal dependencies, we adopt a teacher-forcing strategy to handle its recurrent hidden states during training. Both explicit-interface baselines use the same vision encoder architecture. Furthermore, all three methods are trained using identical action supervision and learning-rate schedules to ensure a fair comparison.

As shown in the table below, our latent-interface approach consistently achieves higher mean success rates than both explicit-interface baselines with fewer training updates, across all three dataset sizes and both pose tolerances. Moreover, it exhibits lower variance across random seeds than explicit transfer in all six evaluation settings, demonstrating greater training stability and data efficiency.

Brush alignment success (%) by training episodes (Eps.) and adaptation steps. Values are mean ± SD across five seeds, using each seed’s best evaluation within the stated budget. The explicit teacher is SimToolReal.

Eps.ModelSteps1.5 cm5 cm
200Latent (ours)50k1.7 ± 0.322.4 ± 1.3
200Explicit RL prior100k0.9 ± 0.614.0 ± 8.1
200Explicit teacher200k0.0 ± 0.01.0 ± 1.8
1,000Latent (ours)100k15.9 ± 0.650.7 ± 0.2
1,000Explicit RL prior200k4.4 ± 3.626.7 ± 12.5
1,000Explicit teacher200k0.1 ± 0.28.7 ± 7.1
4,000Latent (ours)200k36.9 ± 1.359.4 ± 1.1
4,000Explicit RL prior240k27.1 ± 9.952.7 ± 8.6
4,000Explicit teacher240k3.7 ± 5.224.7 ± 33.8

We further investigate whether the dimensionality of the interface between the vision encoder and action decoder constitutes a performance bottleneck. In our latent-interface setting, the interface consists of two 384-dimensional tokens, yielding a total dimensionality of 768. In contrast, both the RL Prior (Explicit) and Teacher (Explicit) baselines use a 31-dimensional interface comprising privileged state information. To isolate the effect of interface dimensionality, we introduce a bottleneck module between the vision encoder and action decoder that first projects the 768-dimensional latent representation into an rr-dimensional space and then projects it back to 768 dimensions. We train each configuration using 1,000 episodes with five random seeds. As shown in the table below, reducing the interface dimensionality leads to a decrease in performance. Nevertheless, our approach continues to substantially outperform the baselines, while exhibiting significantly lower variance across training seeds. These findings suggest that the choice of interface plays a critical role in effectively leveraging an RL prior using only action supervision, and that designing such an interface is nontrivial. Our proposed latent interface offers a simple yet effective solution, enabling robust and stable knowledge transfer from the RL prior.

Brush alignment success (%) by interface dimension rr, using 1,000 training episodes. Values are mean ± SD over five seeds.

rr1.5 cm5 cm
80.5 ± 0.29.0 ± 2.2
164.9 ± 1.131.5 ± 2.2
31 (explicit RL prior)4.4 ± 3.626.7 ± 12.5
31 (explicit teacher)0.1 ± 0.28.7 ± 7.1
3210.8 ± 1.143.7 ± 0.9
6413.2 ± 1.545.8 ± 2.0
12814.7 ± 0.748.5 ± 0.9
25614.8 ± 1.348.8 ± 1.7
768 (bottleneck)15.2 ± 0.549.5 ± 0.8
768 (ours, no bottleneck)14.7 ± 0.749.6 ± 0.9

3. RL prior training cost

We reuse the publicly released Isaac Gym teacher from SimToolReal [1], whose authors report approximately 12 days of training on a single A6000 GPU (288 GPU-hours) for their final training run. We further fine-tune this teacher on procedural-object pose alignment using MuJoCo Warp for 500 epochs with 18,432 parallel environments since our task simulation is MuJoCo. This fine-tuning takes approximately 2.5 hours on a single RTX PRO 6000 GPU (2.5 GPU-hours), improving the teacher's success rate from 57% to 80.4%. We subsequently collect 100,000 successful state–action rollouts and distill a single unified action decoder shared across all four downstream tasks: object alignment, door lift, bimanual assembly, and grasp anything.

We emphasize that the primary objective of this work is to investigate whether large-scale RL pretraining can improve the data efficiency of downstream visual policy learning. While training a large-scale RL prior can incur substantial computational costs, collecting visual demonstrations is generally more challenging and resource-intensive than generating simulated RL experience. Our approach aims to leverage the knowledge acquired through large-scale RL to reduce the amount of visual demonstration data required for downstream tasks.

[1] Kedia, Kushal, et al. "Simtoolreal: An object-centric policy for zero-shot dexterous tool manipulation." arXiv preprint arXiv:2602.16863 (2026).

4. Real-world experiments

We agree that the absence of real-robot experiments limits our conclusions. SimToolReal’s policy commands a KUKA iiwa in joint space, and our platform uses a different arm. Testing on our robot therefore requires calibrating the simulator to our physical arm and hand, repeating the time-intensive RL training, and distilling a new action decoder.

We expect the main sim-to-real challenge to lie in the large scale RL training, not in latent interface (one of our main contributions). We would train the latent interface directly on real images and demonstrated actions, so we expect visual domain shift to play a smaller role. If the simulator captures the physical robot’s motion and contact dynamics accurately enough, we expect our approach can enable data-efficient dexterous manipulation learning similar to simulation results. We are working on real world experiments but afraid that we cannot finish them in the rebuttal phase.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.