acceptodds
Under review as a conference paper at ICLR 2027

INVAR: Randomization-Invariant Visual Representations for Zero-Shot Sim-to-Real

Abstract

Humans know to act consistently across real and simulated environments because they can abstract away visual appearance to what matters for the task. Can robots achieve similar visual invariance? We present INVAR, a visual representation for zero-shot sim-to-real transfer. While classic sim-to-real methods rely on task-specific domain randomization (DR) to train visual encoders and action policies, INVAR distills diverse simulation randomizations into a unified, invariant visual encoder that is self-supervised and scalable. It leverages self-distillation over an unlabeled corpus of simulated play data and robot rollouts without requiring real-world images during representation learning. A policy trained purely on nominal simulation data with a frozen INVAR encoder transfers zero-shot to the real world. INVAR nearly doubles the success rate of identical policies built on state-of-the-art pretrained encoders trained with standard DR, across both dexterous manipulation (0.61 vs. 0.30 mean success) and parallel-gripper manipulation (0.65 vs. 0.32). These results demonstrate that decoupling visual invariance into a task-agnostic encoder unlocks scalable, zero-shot sim-to-real transfer, similar to humans. Our website is available at [https://iclrinvar.github.io/invar/](https://iclrinvar.github.io/invar/).

Primary area:
applications to robotics, autonomy, planning
Keywords:
sim-to-real transfer, visual representation learning, robot manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eWqxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

INVAR learns visual invariance from paired clean and randomized simulation renders. Its frozen encoder enables policies trained on nominal simulation to transfer zero-shot to real robots, substantially improving dexterous and bimanual manipulation success over pretrained-encoder baselines.

Critical strengths
  1. Reusable visual robustness. Learning invariance during encoder pretraining allows downstream policies to share the same robust representation without visual randomization during policy training. This addresses a practical bottleneck in sim-to-real learning.
  2. Substantial real-world gains. Success improves from 0.35 to 0.62 on dexterous grasping and from 0.32 to 0.65 on bimanual tasks. Evaluation across two platforms and held-out objects and task concepts provides persuasive evidence of practical value.
Critical weaknesses

All downstream evaluations use task-specific BC policies, but demonstration counts, state-distribution coverage, and policy validation procedures are insufficiently documented. The “unseen” objects and tasks are held out from encoder pretraining, while their downstream policies still receive task-specific demonstrations. Consequently, the experiments do not sufficiently establish whether the gains reflect broadly transferable visual invariance or improved fitting within restricted demonstration distributions. Performance across demonstration budgets, controlled shifts in initial states, and independent training seeds would materially strengthen this assessment.

The symmetric-DR baseline uses crops from one randomized render, whereas INVAR aligns different appearances of the same state. Thus, the comparison supports paired appearance alignment but does not establish that a clean teacher is necessary.

The statistical support and generalization scope is limited. Non-grasping main evaluations use 24 real-world trials per method, while ablations report only 12 rollouts, without uncertainty estimates or clear variability across independently trained models. This weakens conclusions based on smaller differences. Both evaluated embodiments also appear in encoder pretraining, leaving transfer to unseen embodiments untested.

Questions for authors
  1. How many independently trained encoders and policies contribute to each real-world result?
  2. How does INVAR compare with teacher and student views independently randomized from the same simulator state, using matched augmentations and compute? This would distinguish the contribution of paired appearance alignment from that of a clean teacher and directly test the central claim.
4 comments

We thank the reviewer for their thoughtful feedback. We are glad that the reviewer recognizes the practical value of reusable visual robustness and the strong real-world results. We address their questions and concerns below.

How many independently trained encoders and policies contribute to each real-world result?

We pretrain one INVAR vision encoder, which is frozen and reused across all tasks and embodiments. For each downstream task, only a new task-specific behavior-cloning action head is trained on top of the frozen encoder. Unlike approaches requiring task-specific encoder fine-tuning or domain randomization, INVAR learns transferable visual invariance once, achieving strong performance across diverse held-out objects and tasks without retraining the encoder.

Clean teacher and matched randomized-view ablation

How does INVAR compare with teacher and student views independently randomized from the same simulator state, using matched augmentations and compute? This would distinguish the contribution of paired appearance alignment from that of a clean teacher and directly test the central claim.

Our ablation suggests that the main benefit comes from learning invariance across paired views, while a canonical teacher target provides additional gains. We ran an ablation where the teacher and student receive independently randomized renders of the same simulator state, while keeping the pretraining data, augmentations, and optimization budget identical (Sec. 4.5; Table 7).

This variant outperforms the best baseline in simulation with visual DR on both dexterous-hand tasks and in the real world on the ABC put-bottle-in-bin task, but remains below INVAR (canonical teacher) in simulation with visual DR on all three tasks and in the real-world bottle task (Table 7). These results suggest that aligning independently randomized views can improve visual invariance, but aligning randomized observations with a clean teacher provides additional benefits, supporting our asymmetric design choice.

MethodGrasp the HammerGrasp the Spray bottleput bottle in bin
Success rate (%) ↑Real-world SRNominal Sim SRSim SR w/ visual DRReal-world SRNominal Sim SRSim SR w/ visual DRReal-world SRNominal Sim SRSim SR w/ visual DR
INVAR9290927694‾\underline{94}76839454
INVAR with independently randomized teacher and student90‾\underline{90}88‾\underline{88}84‾\underline{84}68‾\underline{68}9870‾\underline{70}71‾\underline{71}92‾\underline{92}52‾\underline{52}
Best baseline7088‾\underline{88}743684624692‾\underline{92}54

Why do we need a canonical teacher?

1) Align with downstream inputs: Since downstream policies using INVAR train only on clean simulation images, clean teacher targets encourage randomized observations to map to the same representations used during policy training.

2) Provide a consistent target: The teacher is updated only through EMA, without direct gradients. Clean inputs keep its visual target consistent, encouraging invariance around the nominal representations used by downstream policies.

Statistical evidence and independent model counts

The statistical support and generalization scope is limited. Non-grasping main evaluations use 24 real-world trials per method, while ablations report only 12 rollouts, without uncertainty estimates or clear variability across independently trained models. This weakens conclusions based on smaller differences.

Physical trial counts per object/task: For each method, our main real-world experiments use 25 rollouts per dexterous-hand task/object (five unseen grasping objects and one unseen pouring task, so 150 rollouts in total) and 24 rollouts per bimanual parallel-gripper task. Our additional real-world ablation studies use 12 rollouts per evaluated condition. These trial counts are consistent with the evaluation scale commonly adopted in foundational robotics research, where hardware constraints and evaluation costs often limit real-world trials (e.g. ACT (Zhao et al., 2023; 25 trials), Diffusion Policy (Chi et al., 2023; 20 trials), OpenVLA (Kim et al., 2024; 5–10 trials), π0.5\pi_{0.5} (Black et al., 2025; 10 trials), RobustVLA (2026; 10 trials)).

Uncertainty for the main and ablation results: We report ±1 SE (one standard error) to quantify variability (Figure 4, Tables 4–6). For the embodiment-diversity ablation (Table 4), we ran three random seeds and found that the embodiment-diversity effect is consistent across training runs. For our main real-world experiments, INVAR consistently achieves approximately 2× the success rate of the strongest baseline across different tasks and embodiments, demonstrating substantial performance improvements rather than marginal gains.

Transfer to unseen embodiments

Both evaluated embodiments also appear in encoder pretraining, leaving transfer to unseen embodiments untested.

INVAR transfers to unseen embodiments without additional encoder pretraining or fine-tuning, suggesting that it learns a visual invariance prior that generalizes beyond its pretraining embodiments. We support this with three complementary evaluations: (1) visual sim-to-real transfer on new embodiment; (2) leave-one-out generalization during encoder pre-training; and (3) encoder generalization to new domain-randomized sim embodiment zero-shot. In all settings, the encoder remains frozen, and only a new action head is trained for the target task (Sec. 4.3, Tables 2–3).

1. Visual sim-to-real transfer on new embodiment: KUKA–SHARPA stirring (Sec. 4.3; Table 2). We evaluate the task “pick up the spoon and stir clockwise” with a KUKA iiwa arm (unseen during pretraining) and SHARPA hand (setup shown in Table 2). INVAR and both baselines use the same inputs and identical policy architecture. INVAR outperforms both baselines in simulation and the real world (64 simulation and 25 real-world rollouts per method).

Encoder / trainingReal-world SRNominal Sim SRSim SR w/ visual DR
INVAR (ours, frozen)52%89.1%90.6%
DINOv3 (end-to-end)28%76.6%82.8%
CAIP (frozen)24%54.7%51.6%

2. Leave-one-out generalization during encoder pre-training (dexhand → parallel gripper, Sec. 4.3; Table 3). We use a frozen INVAR encoder pretrained exclusively on dexterous hand data, then test on the bimanual parallel-gripper task “put bottle in bin” on YAM arms. The experiment shows that our learned encoder INVAR transfers to a morphology unseen in pretraining: the dexterous-hand-only encoder outperforms all baselines in the real world despite never seeing parallel-gripper data during encoder pretraining.

On real-camera parallel-gripper observations, the dexhand-only INVAR encoder attends more to task-relevant bottles, bins, and robot arms, and less to backgrounds than DINOv3 and CAIP (Appendix C.3, Fig. 11).

EncoderPolicy trainingReal-world SRNominal Sim SRSim SR w/ visual DR
INVAR (full pretraining, frozen)Clean83%94%54%
INVAR (dexhand-only, frozen)Clean\underline{\mathit{67\\%}}90%\underline{\mathit{52\\%}}
DINOv3 (frozen)Clean21%60%38%
DINOv3 (fine-tuned)DR42%86%44%
DINOv3 (frozen)DR33%80%34%
CAIP (frozen)Clean33%90%52%
CAIP (frozen)DR46%\underline{92\\%}54%

Simulation success is measured over 50 episodes per method (nominal or visual DR), and real-world success over 24 rollouts per method.

3. Zero-shot encoder transfer to an unseen simulated embodiment under visual DR (Sec. 4.3; Table 2). On RoboMimic/robosuite Square and Stack, both the Franka arm and third-person camera viewpoint are unseen during pretraining. With frozen encoders and matched action-head training and evaluation settings, INVAR outperforms DINOv3 and CAIP in simulation with visual DR on both tasks (100 episodes per method).

Frozen encoderSquareStack
Sim SR w/ visual DRNominal Sim SRSim SR w/ visual DRNominal Sim SR
INVAR (ours)76%85%77%83%
DINOv370%83%71%81%
CAIP39%72%49%85%

4. Qualitative cross-embodiment evidence (Sec. 4.4; Appendix C.3, Fig. 10). Across unseen human egocentric views (EgoDex), dexterous hands (Wuji and Allegro), and robot-arm configurations (Franka and KUKA–SHARPA), INVAR shows more object- and interaction-focused attention than DINOv3 and CAIP on identical observations. Wuji cart-pushing and bed-making examples from WB-WAM extend this qualitative analysis beyond tabletop manipulation to humanoid locomanipulation.

5. Scalability of embodiment pretraining. INVAR is designed to scale efficiently to a broad range of robot embodiments for two reasons:

  1. Generalization beyond pretraining embodiments: Our results demonstrate that INVAR transfers to unseen robot configurations without additional paired renders or encoder retraining. Moreover, most manipulation robots fall into a manageable set of embodiment families, such as single-arm or bimanual systems with dexterous hands or parallel grippers, suggesting that a sufficiently diverse pretrained encoder could support many downstream robots.
  2. Efficient expansion of pretraining coverage: Even when additional embodiments need to be included in pretraining, all encoder pretraining data are generated in simulation, allowing us to expand the dataset at low cost without collecting real-world data. Visual robustness can then be learned once and reused across downstream tasks and embodiments, rather than requiring task-specific domain randomization and representation learning for every policy.

Demonstration coverage and controlled variation

The “unseen” objects and tasks are held out from encoder pretraining, while their downstream policies still receive task-specific demonstrations. Consequently, the experiments do not sufficiently establish whether the gains reflect broadly transferable visual invariance or improved fitting within restricted demonstration distributions.

Our goal is to evaluate the sim-to-real generalization of a reusable visual encoder, rather than zero-shot transfer of an action policy across tasks. INVAR is pretrained once and then frozen and reused across all downstream tasks and embodiments. Each task has its own behavior-cloning action head, trained using task-specific demonstrations on nominal simulation observations. Importantly, the evaluation objects and tasks are held out from encoder pretraining. Thus, our central contribution is that a single task-agnostic encoder provides transferable visual invariance across previously unseen downstream settings, without requiring task-specific visual domain randomization or encoder retraining. We have added clean-versus-randomized feature visualizations to directly illustrate the learned visual invariance (Sec. 4.4; Appendix C.4, Fig. 12).

All downstream evaluations use task-specific BC policies, but demonstration counts, state-distribution coverage, and policy validation procedures are insufficiently documented. Performance across demonstration budgets, controlled shifts in initial states, and independent training seeds would materially strengthen this assessment.

Demonstration counts, state-distribution coverage, and policy validation: We report the demonstration counts and initial-state distributions below and in the revised paper (Appendix B.2, Tables 10–11).

Simulation demonstrations per downstream task.

EmbodimentTask/objectPolicy trainingHeld-out validation
DexhandSpray bottle1,600400
DexhandWine glass1,300320
DexhandWatering can23055
DexhandTape1,100270
DexhandHammer1,600400
DexhandWine pouring400100
Parallel gripperPut bottles in bin1,700190
Parallel gripperFlip mug1,800200

We retain only successful simulation rollouts during demonstration generation. Consequently, demonstration counts vary across tasks due to differences in data-collection success rates.

Initial-state distribution coverage.

Task/objectPosition variationOrientation and additional variation
Spray bottle, tape±5 cm in X/Y±20° yaw
Wine glass, watering can±4 cm in X/Y±10° yaw
Hammer±5 cm in X/Y−10° to +30° yaw
Wine pouringBottle: 5 × 5 cm; bowl: 3 × 5 cm±10° yaw for both
Put bottles in binObject X: 0.319–0.881 m; Y: −0.632–0.633 m19 object geometries; 0.9–1.1× object scale; 360° object yaw. Bin X: 0.57–0.73 m, Y: ±0.25 m; ±43° yaw; 1.0–1.4× scale
Flip mugObject X: 0.469–0.783 m; Y: −0.343–0.347 m18 object geometries; 0.9–1.1× object scale; 360° object yaw subject to valid placement. Tray X: 0.55–0.70 m, Y: ±0.30 m; ±14° yaw

Our initial-state randomization follows established practices in robot manipulation benchmarks. Our ±4–5 cm reset ranges are comparable to standard benchmarks, such as RoboMimic Lift (±3 cm; Mandlekar et al., 2022) and a representative LIBERO-Spatial task (±1–7 cm; Liu et al., 2023), while our parallel-gripper tasks cover larger position ranges, object geometries, and orientations.

Overall, our evaluation spans substantially different demonstration budgets, object geometries, initial-state distributions, and manipulation tasks. INVAR consistently achieves strong performance across these diverse settings, providing evidence that its learned representations are effective beyond a single narrowly constrained demonstration distribution.

Matched comparisons isolate the encoder’s contribution: For each task, INVAR and all baselines use identical demonstrations, initial-state distributions, object variations, policy architectures, and training hyperparameters. These controls attribute the relative performance gains to the visual representations.

Dear Reviewer,

Thank you again for your time and thoughtful feedback. Please let us know if you have any further questions or concerns; we would be very happy to address them.

Reviewer_TwebOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

INVAR moves visual domain randomization from per-task policy learning into a reusable representation-learning stage. A DINOv3-based student processes randomized renderings, while an EMA teacher processes nominal renderings of the same simulator state. The encoder is adapted using a DINO-style self-distillation objective on approximately 6.3 million paired frames spanning dexterous manipulation, bimanual manipulation, and humanoid data. The encoder is then frozen, and downstream behavior-cloning policies are trained on nominal simulation observations. On held-out objects and tasks, the paper reports substantial real-world gains over DINOv3 and CAIP baselines, including mean dexterous grasping success of 0.62 versus 0.35 for CAIP with domain-randomized policy data, and mean bimanual success of 0.65 versus 0.32 for the strongest corresponding DINOv3 baseline. Additional experiments examine representation properties, pretraining diversity, objectives, and backbones.

Critical strengths
  1. The physical-transfer improvements are substantial and cover two different manipulation platforms. The gains are not restricted to one object or to nominal simulation. Table 7 reports improvements across the five grasping objects, and Table 8 reports 0.83 versus 0.42 on bottle-in-bin and 0.46 versus 0.21 on mug flipping against the finetuned DINOv3/DR configuration. The encoder is reused while downstream policies are trained without visual DR, which makes the result practically meaningful. Section 3.2 explicitly states that the downstream evaluation objects and task concepts are excluded from the relevant encoder-pretraining corpus.

  2. The analysis addresses plausible alternative explanations. The baseline grid varies encoder finetuning and policy-training randomization; the symmetric-DR comparison matches pretraining resources; and the ConvNeXt result tests whether the recipe extends beyond the main Transformer backbone. The content-gap control in Table 9 is particularly useful: it distinguishes genuine clean–randomized correspondence from a degenerate position-code solution. Together, these experiments support a representation-learning contribution beyond simply adding more downstream policy data.

Critical weaknesses
  1. The main objective ablation does not isolate the value of a canonical teacher from the value of cross-render pairing. In Section 4.4, the symmetric-DR baseline derives both teacher and student views from the same randomized render, whereas INVAR uses two different renderings of a shared state. Thus, the 0.84 versus 0.42 comparison changes both the teacher's canonical appearance and whether training enforces consistency across rendering conditions. It supports the benefit of the complete paired-render recipe, but does not establish that a clean teacher is essential. A matched control with two independently randomized renders of the same state would make the specific mechanism substantially clearer.

  2. The precision and resource interpretation of the secondary claims remain uncertain. The main evaluations use limited physical rollouts, and Table 2 reports only 12 rollouts for ablations; the aggregation across the two dexterous tasks and training seeds is not fully specified. Confidence intervals are absent, so small differences such as clean-policy versus DR-policy training should not be read as evidence of equivalence. Table 1 measures epochs to each method's own peak, rather than time to a common success threshold, and excludes the one-time encoder-training cost. Appendix A also uses action labels for checkpoint selection via a linear probe: this is compatible with self-supervised encoder gradients, but the labeled-data and validation requirements should be made explicit when presenting the method as label-free and reusable.

Questions for authors
  1. How does INVAR compare with a teacher and student receiving independently randomized renderings of the same simulator state, under the same data and optimization budget? This would distinguish the benefit of paired-state invariance from the additional benefit of a canonical target.

  2. Please specify physical trial counts per object/task, the number of independently trained policies, how the dexterous ablation scores are aggregated, and uncertainty for the main and ablation results. Does the embodiment-diversity effect remain consistent across training runs?

  3. Which episodes, tasks, and action labels are used for fitting and evaluating the checkpoint-selection probes? Please confirm their separation from downstream evaluation and explain how a single checkpoint is selected across domains. How sensitive are the results to using the final checkpoint instead?

3 comments

We thank the reviewer for their careful and constructive review. We are pleased that the reviewer finds the real-world gains substantial and the controlled comparisons informative. We address their questions and concerns below.

Clean teacher and matched randomized-view ablation

How does INVAR compare with a teacher and student receiving independently randomized renderings of the same simulator state, under the same data and optimization budget? This would distinguish the benefit of paired-state invariance from the additional benefit of a canonical target.

Our ablation suggests that the main benefit comes from learning invariance across paired views, while a canonical teacher target provides additional gains. We ran an ablation where the teacher and student receive independently randomized renders of the same simulator state, while keeping the pretraining data, augmentations, and optimization budget identical (Sec. 4.5; Table 7).

This variant outperforms the best baseline in simulation with visual DR on both dexterous-hand tasks and in the real world on the ABC put-bottle-in-bin task, but remains below INVAR (canonical teacher) in simulation with visual DR on all three tasks and in the real-world bottle task (Table 7). These results suggest that aligning independently randomized views can improve visual invariance, but aligning randomized observations with a clean teacher provides additional benefits, supporting our asymmetric design choice.

MethodGrasp the HammerGrasp the Spray BottlePut bottle in bin
SR (%) ↑Real-world SRNominal Sim SRSim SR w/ visual DRReal-world SRNominal Sim SRSim SR w/ visual DRReal-world SRNominal Sim SRSim SR w/ visual DR
INVAR9290927694‾\underline{94}76839454
INVAR with independently randomized teacher and student90‾\underline{90}88‾\underline{88}84‾\underline{84}68‾\underline{68}9870‾\underline{70}71‾\underline{71}92‾\underline{92}52‾\underline{52}
Best baseline7088‾\underline{88}743684624692‾\underline{92}54

Why do we need a canonical teacher?

1) Align with downstream inputs: Since downstream policies using INVAR train only on clean simulation images, clean teacher targets encourage randomized observations to map to the same representations used during policy training.

2) Provide a consistent target: The teacher is updated only through EMA, without direct gradients. Clean inputs keep its visual target consistent, encouraging invariance around the nominal representations used by downstream policies.

Evaluation clarification

Please specify physical trial counts per object/task, the number of independently trained policies, how the dexterous ablation scores are aggregated, and uncertainty for the main and ablation results. Does the embodiment-diversity effect remain consistent across training runs?

Physical trial counts per object/task: For each method, our main real-world experiments use 25 rollouts per dexterous-hand task/object (five unseen grasping objects and one unseen pouring task, so 150 rollouts in total) and 24 rollouts per bimanual parallel-gripper task. Our additional real-world ablation studies use 12 rollouts per evaluated condition. These trial counts are consistent with the evaluation scale commonly adopted in foundational robotics research, where hardware constraints and evaluation costs often limit real-world trials (e.g. ACT (Zhao et al., 2023; 25 trials), Diffusion Policy (Chi et al., 2023; 20 trials), OpenVLA (Kim et al., 2024; 5–10 trials), π0.5\pi_{0.5} (Black et al., 2025; 10 trials), RobustVLA (2026; 10 trials)).

Number of independently trained policies: One shared INVAR encoder is reused across tasks, while each downstream policy/action head is trained for its own task. This task-agnostic reusability is one of our key contributions: INVAR learns transferable visual representations once, eliminating the need to retrain or fine-tune the visual encoder for each new task.

Dexterous-hand ablation aggregation: We report the average success rate across the two dexterous-hand ablation tasks, with 12 real-world trials per task per method per seed.

Uncertainty for the main and ablation results: We report ±1 SE (one standard error) to quantify variability (Figure 4, Tables 4–6). For the embodiment-diversity ablation (Table 4), we ran three random seeds and found that the embodiment-diversity effect is consistent across training runs. For our main real-world experiments, INVAR consistently achieves approximately 2× the success rate of the strongest baseline across different tasks and embodiments, demonstrating substantial performance improvements rather than marginal gains.

Continued in Part 2 below.

Continued from Part 1.

Table 1 measures epochs to each method's own peak, rather than time to a common success threshold, and excludes the one-time encoder-training cost.

Relative downstream policy convergence speed is measured by the epochs required to reach 90% of each method's own peak success rate (Table 1), separating convergence speed from final performance. Because INVAR achieves a higher peak success rate than the baselines, its 90%-of-peak criterion is also a higher absolute target, yet INVAR reaches it earlier.

We further evaluate convergence to common absolute success thresholds (Appendix C.2, Table 15), wall-clock convergence times (Appendix C.2, Table 16) and plot corresponding learning curves (Appendix C.2, Fig. 9). Together, these results support faster downstream policy learning with INVAR.

Convergence to common absolute success thresholds (epochs; lower is faster, Appendix C.2, Table 15).

MethodSR ≥ 0.40SR ≥ 0.45SR ≥ 0.50
INVAR · frozen, clean182527
DINOv3 · frozen, clean———
DINOv3 · fine-tuned, DR59——
DINOv3 · frozen, DR5091134
CAIP · frozen, clean77——
CAIP · frozen, DR65109—

Policy convergence speed. Wall-clock hours to reach 90% of each method's own peak success rate (lower is faster; Appendix C.2, Table 16).

EvaluationINVAR (frozen, clean)DINOv3 (frozen, clean)DINOv3 (fine-tuned, DR)DINOv3 (frozen, DR)CAIP (frozen, clean)CAIP (frozen, DR)
Sim nominal3.16.211.36.518.522.0
Sim w/ visual DR2.95.38.011.813.826.3

Amortized encoder pretraining cost: Table 1 excludes one-time encoder pretraining, which takes approximately 8 hours on 8 L40S GPUs (64 GPU-hours) to fine-tune DINOv3. Reusing the frozen encoder across N tasks reduces its per-task cost as 1/N, while every task benefits from clean-simulation training and faster policy convergence.

Linear probe labels and checkpoint selection

Which episodes, tasks, and action labels are used for fitting and evaluating the checkpoint-selection probes? Please confirm their separation from downstream evaluation.

Checkpoint-selection data and evaluation separation: We use action-labeled grasping episodes from 20 dexterous-hand pretraining objects to fit a linear probe predicting 28-D simulator actions. For each object, 32 episodes are used for fitting and eight for validation, with six frames sampled per episode (800 episodes and 4,800 frames in total for 20 objects).

These data are entirely separate from downstream evaluation objects and episodes, which include grasping five unseen objects, a novel pouring task, and parallel-gripper manipulation tasks.

Explain how a single checkpoint is selected across domains.

Checkpoint selection uses only dexterous-hand grasping data, the same selected checkpoint is reused across all embodiments without further encoder training or checkpoint selection. INVAR achieves strong performance on parallel-gripper tasks despite their absence from checkpoint selection, demonstrating transfer beyond the selection domain and suggesting that the method is robust to the choice of selection domain.

How sensitive are the results to using the final checkpoint instead?

Not sensitive. The final (24k) checkpoint performs comparably to the selected (20k) checkpoint across all three tasks, with success rates differing by at most 4 percentage points. These results suggest that downstream performance is not sensitive to encoder checkpoint selection.

Encoder checkpointGrasp the Spray BottlePour Wine into the BowlABC Put Bottles
Success rate (%) ↑Nominal Sim SRSim SR w/ visual DRNominal Sim SRSim SR w/ visual DRNominal Sim SRSim SR w/ visual DR
20k (selected ckpt)88%56%65%65%94%76%
24k (final ckpt)92%54%65%65%94%80%

Appendix A also uses action labels for checkpoint selection via a linear probe: this is compatible with self-supervised encoder gradients, but the labeled-data and validation requirements should be made explicit when presenting the method as label-free and reusable.

No action labels are required for self-distillation. Action labels are used only for checkpoint selection and never contribute gradients to the visual encoder. These details, along with the content below, have been added to Appendix B.3 of the revised paper.

Dear Reviewer,

Thank you again for your time and thoughtful feedback. Please let us know if you have any further questions or concerns; we would be very happy to address them.

Reviewer_KJzyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how to decouple visual robustness from policy learning for zero-shot sim-to-real transfer. The authors argue that with policy-level domain randomization (DR), each policy must learn control while also learning to ignore task-irrelevant visual features, and this invariance is discarded with the policy. To address this, INVAR fine-tunes a DINOv3 encoder by self-distillation on about 6.3M pairs of clean and randomized renders of the same simulator states across three embodiments, where an EMA teacher sees the clean render and the student sees the randomized one. The frozen encoder is then used to train behavior cloning policies on clean simulation data only. Real-world experiments on a dexterous hand and a bimanual parallel-gripper setup show that INVAR roughly doubles the success rate of policies built on DINOv3 and CAIP trained with DR, on objects and tasks unseen during pretraining. Furthermore, this work ablates the pretraining data, the distillation objective, and the backbone, and analyzes the learned representations.

Critical strengths
  • While classic sim-to-real pipelines randomize visual appearance during policy training, this work moves randomization into a one-time pretraining stage of a task-agnostic encoder, using clean and randomized renders of the same simulator state as exact positive pairs instead of heuristic image augmentations. With the frozen encoder, policies trained only on clean simulation data outperform every baseline on all eight real-world tasks and objects (Tables 7, 8), reaching 0.62 success on unseen-object grasping versus 0.35 for the strongest baseline, and 0.65 versus 0.32 on the bimanual tasks (Figure 4).
  • The ablations help isolate where the transfer gains come from. A symmetric variant trained on the same randomized data performs on par with INVAR in nominal simulation but reaches only half its real-world success (Table 4), suggesting that the gains come from aligning randomized views to clean ones rather than from exposure to randomized data. Moreover, with the frozen INVAR encoder, adding DR to the policy’s training data brings no further benefit (0.79 vs 0.84), which supports the claim that visual invariance can be learned once rather than relearned by each policy.
Critical weaknesses
  • The claim that visual randomization becomes a one-time pretraining cost (Sections 1 and 4.4) is only tested on embodiments seen during pretraining. Section 3.2 states that the evaluation covers novel objects and tasks on the same embodiments used for pretraining, and the humanoid domain is never evaluated downstream. It therefore remains unclear whether a frozen INVAR encoder can support policy learning for an unseen robot, or whether each new embodiment requires new paired renders and re-pretraining, which would make the cost per embodiment rather than one-time. This matters all the more because the paper identifies embodiment diversity as the critical driver of physical generalization (Table 2, Section 5).
  • The two main ingredients, paired renders of the same simulator state and decoupling visual invariance from policy learning, both have precedent that the paper does not discuss. RCAN (James et al., 2019) learns to translate randomized renders into canonical ones, so that a grasping policy trained in a canonical simulator transfers zero-shot to the real world. SODA (Hansen & Wang, 2021) aligns augmented observations to non-augmented ones through an EMA target encoder while training the policy only on non-augmented data, motivated by the same burden on policy learning. Clarifying what INVAR adds beyond these, such as representation-level alignment of a foundation encoder shared across embodiments and tasks, would help assess the novelty of reframing DR as self-supervision (Section 1).
Questions for authors
  • Do the symmetric-DR ablation and the DR data used for the baseline policies include the segmentation-guided background swaps with real photographs and the photometric augmentations that INVAR’s student crops receive (Appendix A)? Table 10 labels the symmetric variant as “crop-only views”, whereas Section 4.4 describes cropping with photometric augmentations. If these differ, part of the gains in - - Figure 4 and Table 4 may come from the augmentations rather than the asymmetric objective. Could the authors evaluate INVAR on an embodiment absent from pretraining, for example by pretraining without the parallel-gripper data and training the bimanual policies on the resulting frozen encoder? Evidence that INVAR transfers to unseen embodiments without new paired renders would raise my assessment.
3 comments

We thank the reviewer for their thoughtful and constructive review. We are delighted that the reviewer appreciates INVAR’s reusable representations, real-world gains, and supporting ablations. We address their questions and concerns below.

Transfer to unseen embodiments

It remains unclear whether a frozen INVAR encoder can support policy learning for an unseen robot, or whether each new embodiment requires new paired renders and re-pretraining, which would make the cost per embodiment rather than one-time.

Could the authors evaluate INVAR on an embodiment absent from pretraining, for example by pretraining without the parallel-gripper data and training the bimanual policies on the resulting frozen encoder? Evidence that INVAR transfers to unseen embodiments without new paired renders would raise my assessment.

INVAR transfers to unseen embodiments without additional encoder pretraining or fine-tuning, suggesting that it learns a visual invariance prior that generalizes beyond its pretraining embodiments. We support this with three complementary evaluations: (1) visual sim-to-real transfer on new embodiment; (2) leave-one-out generalization during encoder pre-training; and (3) encoder generalization to new domain-randomized sim embodiment zero-shot. In all settings, the encoder remains frozen, and only a new action head is trained for the target task (Sec. 4.3, Tables 2–3).

1. Visual sim-to-real transfer on new embodiment: KUKA–SHARPA stirring (Sec. 4.3; Table 2). We evaluate the task “pick up the spoon and stir clockwise” with a KUKA iiwa arm (unseen during pretraining) and SHARPA hand (setup shown in Table 2). INVAR and both baselines use the same inputs and identical policy architecture. INVAR outperforms both baselines in simulation and the real world (64 simulation and 25 real-world rollouts per method). We believe that more accurate simulation physics and larger-scale encoder pretraining could further narrow the sim-to-real gap.

Encoder / trainingReal-world SRNominal Sim SRSim SR w/ visual DR
INVAR (ours, frozen)52%89.1%90.6%
DINOv3 (end-to-end)28%76.6%82.8%
CAIP (frozen)24%54.7%51.6%

2. Leave-one-out generalization during encoder pre-training (dexhand → parallel gripper, Sec. 4.3; Table 3). We use a frozen INVAR encoder pretrained exclusively on dexterous hand data, then test on the bimanual parallel-gripper task “put bottle in bin” on YAM arms. The experiment shows that our learned encoder INVAR transfers to a morphology unseen in pretraining: the dexterous-hand-only encoder outperforms all baselines in the real world despite never seeing parallel-gripper data during encoder pretraining.

We also visualize attention maps from the dexhand-only INVAR encoder on real-camera parallel-gripper observations (Appendix C.3, Fig. 11). Despite never seeing gripper data during pretraining, INVAR shows stronger attention to task-relevant bottles, bins, and robot arms and less background attention than DINOv3 and CAIP, providing insight into the mechanism underlying the model’s generalization.

EncoderPolicy trainingReal-world SRNominal Sim SRSim SR w/ visual DR
INVAR (full pretraining, frozen)Clean83%94%54%
INVAR (dexhand-only, frozen)Clean\underline{\mathit{67\\%}}90%\underline{\mathit{52\\%}}
DINOv3 (frozen)Clean21%60%38%
DINOv3 (fine-tuned)DR42%86%44%
DINOv3 (frozen)DR33%80%34%
CAIP (frozen)Clean33%90%52%
CAIP (frozen)DR46%\underline{92\\%}54%

Simulation success is measured over 50 episodes per method (nominal or visual DR), and real-world success over 24 rollouts per method.

3. Zero-shot encoder transfer to an unseen simulated embodiment under visual DR (Sec. 4.3; Table 2). On RoboMimic/robosuite Square and Stack, both the Franka arm and third-person camera viewpoint are unseen during pretraining. With frozen encoders and matched action-head training and evaluation settings, INVAR outperforms DINOv3 and CAIP in simulation with visual DR on both tasks (100 episodes per method). These visual-DR results, shown first for each task, provide the key evidence that INVAR’s learned visual invariance transfers to unseen tasks and embodiments without encoder adaptation.

Frozen encoderSquare: Sim SR w/ visual DRSquare: Nominal Sim SRStack: Sim SR w/ visual DRStack: Nominal Sim SR
INVAR (ours)76%85%77%83%
DINOv370%83%71%81%
CAIP39%72%49%85%

Continued in Part 2 below.

Continued from Part 1.

4. Qualitative cross-embodiment evidence (Sec. 4.4; Appendix C.3, Fig. 10). Across unseen human egocentric views (EgoDex), dexterous hands (Wuji and Allegro), and robot-arm configurations (Franka and KUKA–SHARPA), INVAR shows more object- and interaction-focused attention than DINOv3 and CAIP on identical observations. Wuji cart-pushing and bed-making examples from WB-WAM extend this qualitative analysis beyond tabletop manipulation to humanoid locomanipulation.

5. Scalability of embodiment pretraining. INVAR is designed to scale efficiently to a broad range of robot embodiments for two reasons:

  1. Generalization beyond pretraining embodiments: Our results demonstrate that INVAR transfers to unseen robot configurations without additional paired renders or encoder retraining. Moreover, most manipulation robots fall into a manageable set of embodiment families, such as single-arm or bimanual systems with dexterous hands or parallel grippers, suggesting that a sufficiently diverse pretrained encoder could support many downstream robots.
  2. Efficient expansion of pretraining coverage: Even when additional embodiments need to be included in pretraining, all encoder pretraining data are generated in simulation, allowing us to expand the dataset at low cost without collecting real-world data. Visual robustness can then be learned once and reused across downstream tasks and embodiments, rather than requiring task-specific domain randomization and representation learning for every policy.

Relation to prior work (RCAN and SODA)

Clarifying what INVAR adds beyond these, such as representation-level alignment of a foundation encoder shared across embodiments and tasks, would help assess the novelty of reframing DR as self-supervision (Section 1).

We acknowledge the reviewer’s comments and these are addressed in the revised related work section 2.2. In brief:

Distinction from RCAN: RCAN translates randomized observations into canonical images and requires a generator at test time. INVAR instead learns a reusable visual representation and does not require a test-time image-translation generator.

Distinction from SODA: SODA learns visual invariance through an auxiliary consistency loss during each task's policy training, so invariance is learned on a per-task basis. INVAR instead pretrains a task-agnostic encoder without task supervision, then freezes and reuses it across downstream tasks and embodiments.

INVAR's central distinction: The encoder is trained once at the representation level and reused across downstream tasks and embodiments, while the task-specific action heads can be trained separately on clean simulation data. This eliminates per-task visual adaptation, enables policy training on clean simulation data, and accelerates downstream convergence.

Matched domain randomization and augmentations

Do the symmetric-DR ablation and the DR data used for the baseline policies include the segmentation-guided background swaps with real photographs and the photometric augmentations that INVAR’s student crops receive (Appendix A)? Table 10 labels the symmetric variant as “crop-only views”, whereas Section 4.4 describes cropping with photometric augmentations. If these differ, part of the gains in - - Figure 4 and Table 4 may come from the augmentations rather than the asymmetric objective.

Symmetric-DR ablation clarification: The teacher and student share the same background swap (either neither has a background swap, or both use the same swapped background). However, their views differ through cropping and photometric augmentations, both of which are used in the symmetric variant. Therefore, the original Table 10’s “crop-only views” label actually refers to cropping with photometric augmentations. We have clarified this in the revised paper, and thank the reviewer for pointing out the ambiguity. We designed this ablation to distinguish the benefit of learning visual invariance through INVAR’s asymmetric clean-to-randomized alignment from simply exposing the model to randomized images.

Matched DR for baseline policies: Yes. The DR-trained baselines use the same segmentation-guided photographic background swaps and photometric augmentations as INVAR's randomized student views. Thus, the comparison is fair with respect to visual randomization, and INVAR's gains cannot be attributed to stronger augmentations.

We have clarified the matched DR settings in Appendix B.2 and Section 4.5, and corrected the symmetric-DR label and description in Table 6 of the revised paper.

Dear Reviewer,

Thank you again for your time and thoughtful feedback. Please let us know if you have any further questions or concerns; we would be very happy to address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.