acceptodds
Under review as a conference paper at ICLR 2027

Pretraining Foundation Policies for Perceptive Humanoid Locomotion with Offline RL

Abstract

Online reinforcement learning has enabled perceptive humanoids to traverse challenging terrain and to transfer these skills to real hardware. Scaling this paradigm across terrains and robot embodiments remains costly, because each new setting requires substantial environment interaction and careful tuning. Recent approaches reduce this cost in two stages. They first train specialist policies online, then distill their rollouts offline into a more general policy through behavior cloning (BC). BC, however, ignores the recorded returns and imitates every behavior equally, which limits what it can learn from the rollouts of experts of varying quality. We study whether perceptive humanoid locomotion policies that span terrains and embodiments can be learned entirely from such offline data. To this end, we introduce Segment-level Offline Imitation of Diverse Experts (SOFIE), an offline reinforcement learning approach based on advantage-weighted regression that uses the recorded rewards to estimate footstep-level advantages and preferentially imitates the better footsteps. Quantile normalization of the advantages within each robot–terrain group keeps the resulting weights consistent across robots, terrains and reward scales. Across twelve humanoids in two simulators, SOFIE outperforms behavior cloning and offline reinforcement learning baselines with policies shared across robots or terrains, including one perceptive policy for three humanoids on four terrains. When fine-tuned on a small dataset from an unseen humanoid, a policy pretrained with SOFIE also outperforms training from scratch. These results suggest that perceptive foundation policies for humanoid locomotion can be pretrained from recorded rollouts alone, without further environment interaction.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Offline RL, humanoid locomotion, perceptive locomotion, multi-embodiment learning, foundation policies, generalist policies

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oHEWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper come up with SOFIEm offline learning method for humanoid robots terrain traversal, which estimates segment advantages to truncated Monte Carlo returns, and train a morphology-conditioned policy. In the experiments, authors evaluate multiple humanoids on different terrains, which shows SOFIE improves performance than BC and offline RL.

Critical strengths

1.The segment-weighting idea is well motivated for locomotion

Critical weaknesses
  1. The motivation is not grounded. The humanoid locomotion under perfect height map is been solved well in humanoid domain, and locomotion is so matured with PPO. I can train a policy like this with mjlab default setting + PPO within 3 hours even single gpu. Also the offline dataset with reward, state, contact label is easy to build in simulation but extremely tough to get in real world since you need mocap system, and feet contact estimation are always noisy. Also the dataset expert are actually from PPO, there are already a lot of papers that could one model control multiple robots like Locoformer.

  2. The experiments are not convincing. Evaluation is same in distribution terrain level with training. Lack strong offline RL benchmark and baseline like decision transformer,ReBRAC...... Yet there is no wall-clock or GPU-hour comparison against online training or an expert-distillation pipeline. These make it unclear when the proposed pipeline offers a meaningful advantage beyond its fixed-dataset evaluation.

Questions for authors

Without an expert-only BC baseline in the perceptive experiments, what supports the claim that selectively learning from mixed-quality data is preferable to simply distilling the converged experts already available?

Since Table 15 compares SOFIE pretraining only against random initialization, what evidence attributes the transfer gains to SOFIE rather than to pretraining itself?

Given that the pipeline requires online-trained experts and substantial offline optimization, what's the advantage over online method?

Could you eval your method in a more standard and solid benchmark like OG bench? they also have env like humanoid and ant. And compare with SOTA offline RL/BC/goal condition BC policy

In general I think both motivation and experiments still need large improvement, I am really concerned that authors can't resolve my concern during rebuttal.

6 comments
AuthorsOpenReview ↗

[post 1/2] Learning setting and observations (W1)

We thank the reviewer for finding segment-level advantage weighting well motivated. However, we respectfully disagree with the assessment that "the motivation is not grounded" and that "the experiments are not convincing." Several concerns treat the task as training an individual locomotion specialist, while the paper learns one policy shared across robots and terrains from recorded rollouts. We address each point below.

"I can train a policy like this with mjlab default setting + PPO within 3 hours even single gpu."

The mjlab default example (Zakka et al., 2025) trains G1 to follow velocity commands on flat ground, and we likewise use online RL to train each specialist for one robot-terrain pair (Appendix B.5). Our question begins after that step: how the recorded rollouts of these specialists can be used to learn one perceptive policy across three humanoids and four terrains. Combining the skills is hard, because RL on several terrains tends to focus on a subset of them and never find a solution for the others (Rudin et al., 2026, Sec. 4.1), which is why recent approaches distill terrain specialists. Even this route is costly for a single robot: RPL trains its perceptive G1 terrain experts for 24 hours on four GPUs and distills them for another 12 hours on eight GPUs (Zhang et al., 2026, Sec. IV-A).

"[...] the offline dataset with reward, state, contact label is easy to build in simulation but extremely tough to get in real world since you need mocap system, and feet contact estimation are always noisy."

Segment-level weighting needs temporal boundaries, which our tasks provide through the gait clock (Practical implementation, Section 4.2). These boundaries require neither contact labels nor motion capture, and the new fixed-window sensitivity analysis supports using any interval that spans at least one footstep (Table 22). The observations, actions and rewards are recorded in simulation during specialist training, so the learning setting does not depend on recovering these quantities from real-world logs.

"Also the dataset expert are actually from PPO, there are already a lot of papers that could one model control multiple robots like Locoformer."

To clarify this, we refer the reviewer to the Introduction and the Related Work (Section 2). We note that none of the multi-robot policies discussed there perceives the terrain. URMA omits exteroceptive sensors (Bohlinger et al., 2024, Sec. 3), GenBot-1K is proprioceptive and studies flat terrain only (Ai et al., 2025), and LocoFormer acts on observations in a unified joint space, without a height map, depth or camera (Liu et al., 2025, Sec. 2.1). Perception matters on our terrains. To evaluate the importance of the perceptive module, we replace the height scan at test time with the average of all training scans (rather than zeros, to keep the input within the range seen in training). Our fine-tuned policies then lose more than half of their score on stairs down and almost all of it on rough ground (new, Table 28). In contrast, SOFIE learns one perceptive policy that controls several humanoids on stairs and stepping stones, entirely from recorded rollouts of mixed quality. We additionally evaluate BC following the recipe of GenBot-1K (Experiments, Section 5) and SOFIE with URMA's network (new, Table 29).

"The humanoid locomotion under perfect height map is been solved well in humanoid domain [...]"

We respectfully disagree. To the best of our knowledge, one perceptive policy that controls several humanoid embodiments across challenging terrains has not been learned before. The Related Work (Section 2) reviews the two closest lines of work, perceptive policies across terrains for a single robot (Rudin et al., 2026; Zhang et al., 2026) and policies shared across robot designs (Bohlinger et al., 2024; Ai et al., 2025). Even single-robot specialists on one terrain remain imperfect: ours still fall in 2.9 to 10.6% of episodes on stepping stones and rough ground at the hardest level (Table 5), and RL on several terrains tends to cover only a subset of them (Rudin et al., 2026, Sec. 4.1). A new experiment also shows that SOFIE does not depend on a height map. With every dataset recorded again from a depth camera, SOFIE scores 0.87 compared with 0.49 for BC and leads on every robot-terrain pair with either sensor (Figure 1). The new sim-to-sim experiments go further and set the observations close to those on the robot, using an elevation map built from simulated range scans under drifting odometry or noisy, delayed depth images. After fine-tuning, the policies complete 28 or 29 of 30 stair flights (Table 16).

AuthorsOpenReview ↗

[post 2/2] Evaluation (W2) and questions

W2, evaluation beyond training conditions

Because the main evaluations match the training conditions, the new Table 24 tests whether the performance differences persist when those conditions change. We evaluate the per-terrain policies on harder terrain, under random pushes and with observation noise, and normalize the scores under each condition with the corresponding expert and undertrained checkpoints.

conditionSOFIEIQLBC
training conditions0.880.580.20
terrain level 9 (three terrains)0.780.520.12
random pushes0.720.530.19
observation noise0.360.170.18

SOFIE has the highest mean score under each shift. Observation noise causes the largest drop for SOFIE and IQL, consistent with its absence from the recorded rollouts.

W2, baselines

The baselines cover BC and the offline RL methods relevant to these datasets, namely IQL, goal-conditioned IQL and TD3+BC (Experiments, Section 5). IQL is also the method Abe et al. (2026) use for cross-embodiment pretraining. The revision adds IQL with matched ranking groups and a second policy architecture, so that the effects of weighting and architecture can be examined separately (Tables 20 and 29). We did not evaluate Decision Transformer (Chen et al., 2021) or ReBRAC (Tarasov et al., 2023) and focused the additional experiments on controlled ablations of footstep weighting and quantile normalization (Tables 19 to 22).

W2, wall-clock cost

For the offline stage, SOFIE and BC use the same dataset and actor architecture, and SOFIE additionally fits a value function. Table 7 specifies 100k value-pretraining updates followed by 500k actor updates for the foundation policy, without further environment interaction. The new Figure 12 then measures the cost of subsequent online learning by comparing fine-tuning with training from scratch under the same dense reward. On H1 stairs up, the SOFIE initialization reaches 0.92 after 20M environment steps, while training from scratch reaches that score after 59M steps.

Q1, expert-only BC

Expert-only BC is the reference when converged data can be selected in advance, and the new Figure 10 measures how much of its performance survives when undertrained rollouts are added without quality labels. Expert-only BC scores 0.99, while SOFIE retains 0.90 when half the dataset is undertrained and BC on the same mixed dataset drops to 0.08 (point 2 of the General Response).

Q2, SOFIE or pretraining

We fine-tune the pretrained policies in two settings. With supervised fine-tuning on clean expert rollouts of three new robots (H3-1, T1 and T1VH), SOFIE and BC reach similar returns and IQL slightly lower ones (new rows of Table 15), since all three imitate the same expert data. With new online fine-tuning on the training robots (REPPO, sparse reward, stairs up), learning depends on the policy's own behavior, and the SOFIE foundation, which already starts best, ends highest at 0.83, compared with 0.57 for BC and 0.56 for IQL (Figure 6a).

Q3, comparison with online methods

Online RL trains the specialists, and SOFIE combines their skills offline without further interaction. New experiments then fine-tune the shared SOFIE policy online and compare it with training from scratch under the same reward. The foundation also changes what online RL needs: each specialist was trained with a dense reward of 29 to 36 terms weighted for its robot-terrain pair (Appendix B.4), whereas fine-tuning from the foundation works with a sparse reward that only checks every 2 s whether the robot moved as commanded and penalizes falls. With the experts' dense reward on H1 stairs up, the SOFIE initialization reaches 0.92 using about a third of the environment steps required from scratch (Figure 12), and with a sparse reward on stairs up, training from scratch does not learn while the SOFIE initialization reaches 0.83 (Figure 6a).

Q4, OGBench

Our experiments test footstep weighting and normalization within robot-terrain groups in shared perceptive policies. The humanoidmaze, antmaze and antsoccer tasks of OGBench (Park et al., 2025) study goal-conditioned offline RL with a single embodiment on flat ground, so they do not directly test this combination of policy sharing, challenging terrain and mixed-quality specialist rollouts. We therefore used the additional experiments to isolate the effects of segment weighting, ranking groups and dataset composition (Tables 19 to 23).

We believe these results address each of the reviewer's concerns, and we respectfully ask the reviewer to reconsider the rating in light of them.

AuthorsOpenReview ↗

References

References

Abe et al. (2026). Cross-Embodiment Offline Reinforcement Learning for Heterogeneous Robot Datasets. ICLR.

Ai et al. (2025). Towards Embodiment Scaling Laws in Robot Locomotion. CoRL.

Bohlinger et al. (2024). One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion. CoRL.

Chen et al. (2021). Decision Transformer: Reinforcement Learning via Sequence Modeling. NeurIPS.

Liu et al. (2025). LocoFormer: Generalist Locomotion via Long-context Adaptation. CoRL.

Park et al. (2025). OGBench: Benchmarking Offline Goal-Conditioned RL. ICLR.

Rudin et al. (2026). Parkour in the Wild: Learning a General and Extensible Agile Locomotion Policy Using Multi-Expert Distillation and RL Fine-Tuning. IJRR.

Tarasov et al. (2023). Revisiting the Minimalist Approach to Offline Reinforcement Learning. NeurIPS.

Zakka et al. (2026). mjlab: A Lightweight Framework for GPU-Accelerated Robot Learning. arXiv:2601.22074.

Zhang et al. (2026). RPL: Learning Robust Humanoid Perceptive Locomotion on Challenging Terrains. arXiv:2602.03002.

Reviewer_PCcHOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies offline learning of shared humanoid locomotion policies from trajectories collected by converged and undertrained expert checkpoints. It proposes SOFIE, an advantage-weighted imitation method that assigns a common weight to all transitions within a trajectory segment. In the perceptive locomotion experiments, segments correspond to half-cycles of a prescribed gait clock. Segment advantages are estimated using recorded rewards and a value function trained on truncated Monte Carlo returns, then converted into within-robot–terrain quantile scores to obtain scale-invariant weights. A morphology-conditioned policy is trained through weighted behavior cloning. Validation includes proprioceptive flat-ground locomotion across nine humanoids, perceptive locomotion across G1/H1/M3 and four terrains, and adaptation to three additional robot configurations.

Critical strengths
  1. Substantial improvements in a practically relevant offline learning setting. Learning a shared controller from mixed-quality specialist rollouts is a useful problem, and the improvements are substantial in the most demanding evaluated setting. For the single policy covering three humanoids and four terrains, SOFIE improves normalized tracking from 0.18/0.28 for BC/IQL to 0.68, while reducing falls from 4.3/4.0 to 2.2 per 1,000 steps. Reporting falls separately strengthens the evaluation because tracking reward alone can obscure failures. The comparisons use the same datasets and policy architecture within each setting.
  2. A simple method accompanied by useful temporal-abstraction evidence. SOFIE requires neither additional environment interaction during shared-policy training nor a complicated action-generation mechanism. The segment ablation provides evidence that evaluating multiple actions jointly is useful: on stepping stones, gait-clock segments achieve 0.85, compared with 0.53 for single-step weighting and 0.70 for fixed ten-step segments. This is a concrete, potentially reusable observation about offline learning for contact-rich locomotion.
Critical weaknesses
  1. The experiments do not sufficiently disentangle the central methodological contributions. The method combines established advantage-weighted regression with temporal aggregation and within-group rank normalization. The segment ablation supports temporal aggregation, but the additional benefit of gait-aligned boundaries is task-dependent: on stairs, fixed ten-step segments score 0.72 versus 0.70 for gait-clock segments. Moreover, I did not find a controlled ablation of within-group quantile normalization against global ranking or appropriately scaled raw advantages. The IQL baseline uses whole-batch ranking, whereas SOFIE uses robot–terrain-specific ranking, which further complicates attribution.

  2. Dataset construction and transfer controls limit the broader interpretation of the results. Appendix B.7 selects undertrained checkpoints according to how well their footsteps are separated from expert footsteps by SOFIE’s own advantage score. Although all methods subsequently receive the same dataset, this construction may favor datasets on which the proposed scoring mechanism is especially effective. Robustness to unselected or differently selected mixtures is therefore important. The adaptation experiment also compares SOFIE pretraining only against training from scratch. Table 14 evaluates BC/IQL pretrained policies on the source robots, but does not establish how those initializations perform after equivalent adaptation to the target robots. Thus, the experiment demonstrates a benefit of SOFIE pretraining over no pretraining, but does not isolate a transfer benefit specific to SOFIE.

Questions for authors

Appendix B.7 selects undertrained checkpoints using separation under SOFIE’s segment advantages. How sensitive are the conclusions to this selection procedure? Could you report a controlled comparison using all candidate checkpoints or randomly selected undertrained checkpoints, while keeping dataset size and mixture proportions comparable? This would help determine whether the gains persist beyond datasets selected using the proposed scoring mechanism.

Do you have adaptation results starting from BC- and IQL-pretrained policies trained on the same source data, followed by the same BC fine-tuning protocol on H3-1, T1, and T1VH? If not, could you provide this comparison? It would materially affect whether I interpret the adaptation result as evidence for SOFIE-specific pretraining benefits or primarily as evidence that pretraining is preferable to training from scratch.

2 comments
AuthorsOpenReview ↗

[post 1/2] Weighting ablations (W1)

We thank the reviewer for identifying the comparisons needed to separate the method's components. Appendices E.7 to E.9 add these experiments, and we discuss each concern below.

W1, gait-aligned boundaries

"The segment ablation supports temporal aggregation, but the additional benefit of gait-aligned boundaries is task-dependent [...]"

To separate temporal aggregation from boundary alignment, the new Table 22 compares fixed windows with gait-clock segments in a policy shared across all twelve robot-terrain pairs. The gait clock remains SOFIE's default segment definition, and the fixed windows test how sensitive the weighting is to the interval.

segmentscore
fixed, 10 steps (half a footstep on G1 and H1)0.59
fixed, 20 steps0.68
fixed, 40 steps0.72
gait clock0.66

The window shorter than a footstep performs worse, while windows of one footstep or longer perform comparably to gait-clock segments. We have revised the Results (Section 6) accordingly to state that the weighting interval should span at least one footstep.

W1, ranking groups and raw advantages

"[...] I did not find a controlled ablation of within-group quantile normalization against global ranking or appropriately scaled raw advantages."

Robot-terrain datasets can differ in reward scale and quality, so a shared policy needs weights that remain comparable across groups. We therefore compare within-group ranking with whole-batch ranking and with appropriately scaled raw advantages. In the new Table 19, where the three robots have the same share of undertrained data, whole-batch ranking matches within-group ranking on stairs up at 0.90 but scores lower on stepping stones, at 0.87 compared with 0.93. Raw advantages, with β\beta tuned within each group to match the KL divergence of the ranked weights, score 0.73 on stairs up and 0.85 on stepping stones and fall twice as often on stairs up.

Ranking within pairs matters most when the share of undertrained data differs between robots. The new Table 21 uses twelve robot-terrain pairs with 0% undertrained data for G1, 25% for H1 and 50% for M3. Ranking over the whole batch then shifts weight between robots, whereas ranking within each pair preserves their contributions and raises M3's score from 0.23 to 0.56 and the overall score from 0.60 to 0.73.

W1, IQL with matched ranking groups

"The IQL baseline uses whole-batch ranking, whereas SOFIE uses robot-terrain-specific ranking, which further complicates attribution."

In a new experiment, we rank IQL's advantages either over the whole batch or within each robot-terrain pair (Table 20), to test whether the ranking groups explain SOFIE's advantage. Within-group ranking gives IQL 0.61 on stairs up and 0.54 on stepping stones, compared with 0.62 and 0.48 for whole-batch ranking. SOFIE reaches 0.86 and 0.91 on the same tasks, so the performance gap persists after matching the ranking groups.

AuthorsOpenReview ↗

[post 2/2] Checkpoint selection (Q1) and fine-tuning on new robots (Q2)

Q1 (and W2)

"How sensitive are the conclusions to this selection procedure? Could you report a controlled comparison using all candidate checkpoints or randomly selected undertrained checkpoints, while keeping dataset size and mixture proportions comparable?"

To assess sensitivity to checkpoint selection, we hold dataset size and mixture proportions constant and choose the undertrained checkpoints independently of SOFIE's score. Each dataset in the new Table 23 contains 128 rollouts with 50% converged data, and besides the original selection from Appendix B.7, we sample evenly from all earlier checkpoints or use randomly selected checkpoints for each robot.

undertrained checkpointsstairs up, BC → SOFIEstepping stones, BC → SOFIE
selected as in Appendix B.70.20 → 0.620.28 → 0.84
all earlier checkpoints, sampled evenly0.51 → 0.780.51 → 0.84
randomly selected per robot0.59 → 0.810.49 → 0.85

Both alternative selections preserve SOFIE's advantage over BC, with margins of 0.22 to 0.36. The original selection produces a larger gap because it retains checkpoints whose footsteps differ more from the expert's, which makes the dataset harder for BC. SOFIE also improves on BC when converged rollouts are removed entirely, scoring 0.52 against 0.36 on stairs up and 0.55 against 0.44 on stepping stones.

Q2 (and W2)

"Do you have adaptation results starting from BC- and IQL-pretrained policies trained on the same source data, followed by the same BC fine-tuning protocol on H3-1, T1, and T1VH?"

Yes, in the new rows of Table 15. Policies pretrained with SOFIE, BC and IQL on the same source data (stairs up with G1, H1 and M3) are fine-tuned with the same BC protocol on clean expert rollouts of H3-1, T1 and T1VH. All pretrained policies substantially outperform training from scratch, with SOFIE and BC reaching similar returns and IQL slightly lower ones, as expected when each policy is fine-tuned on the same clean expert data.

The pretraining objective matters where the policy does not receive such clean data. Before any fine-tuning, the pretrained SOFIE policy scores 0.80 on its training robots, compared with 0.66 for IQL and 0.15 for BC (Table 14). In a new online fine-tuning experiment, the foundation policies for G1, H1 and M3 on four terrains, which start at 0.71 (SOFIE), 0.28 (IQL) and 0.15 (BC) on stairs up, are fine-tuned with REPPO and a sparse reward, and SOFIE ends highest at 0.83, compared with 0.57 for BC, 0.56 for IQL and −0.11-0.11 from scratch (Figure 6a, Table 27).

Reviewer_uaFKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes SOFIE, an offline RL learning method for perceptive locomotion policy training that replace BC loss with advantage re-weighted regression using the recorded advantage function so that it can take suboptimal data since the advantage "labels" how good the teacher's action is.

Critical strengths
  1. The paper's approach and the setting and the aim is interesting - pre-training a foundation policy that is purely in an offline setting and turns it into perceptive locomotion policies.
  2. The approach is intuitive and well justified. Since teacher's behavior might not be fully optimal, it is useful to use advantage which contains information of the optimality of the policy, to re-weight the imitation loss.
Critical weaknesses
  1. The major concern I have is the fairness of the comparison. Line 357: "Each robot–terrain pair has a dataset of rollouts, in equal shares from converged and earlier undertrained checkpoints of the same online training run. " How can this be a fair comparison to BC methods which can only imitate the dataset? The basic assumption of teacher makes suboptimal action is probable, but as everyone knows behavior cloning can only imitate what the dataset shows. So this entire set of experiments in the paper only answers the question of can advantage bare label good action out of the bad ones, but cannot tell anything about the proposed method compared to the usual teacher-student distillation paradigm. E.g. dagger style online learning, or IQL on the expert data, or imitation learning methods such as Inverse RL method, or GAIL type of methods.
  2. I cannot verify the robot's performance that claim to achieve perceptive locomotion without deploying on hardware. In this day and age, deployment is just as important, or sometimes even more so, than a fancy solution method. E.g. M3 on stepping stones does not even look close to be deployable. The gait on g1 seems to be highly static on stepping stone and on downstairs it does not adjust the gait at all and I am not convinced this is policy that can work on real robots.
  3. Method wise, the proposed approach is highly restrictive: it requires vast amount of data from an expert, it also requires the reward/advantage it experienced, it then assume the task must be offline (which I don't get because there's already an simulator). The method itself is very specific to the problem setting (bipedal locomotion).
Questions for authors

see weakness.

4 comments
AuthorsOpenReview ↗

[post 1/2] Recorded rewards and expert distillation (W1)

We thank the reviewer for finding the setting interesting and the approach intuitive and well justified.

W1

"How can this be a fair comparison to BC methods which can only imitate the dataset? [...] cannot tell anything about the proposed method compared to the usual teacher-student distillation paradigm. E.g. dagger style online learning, or IQL on the expert data, or imitation learning methods such as Inverse RL method, or GAIL type of methods."

Our comparison trains every method on the same mixed-quality dataset, in which no rollout is labeled as coming from a converged or an undertrained checkpoint. All methods receive the same data and policy architecture, so their differences reflect how they use the recorded experience. To show what such a label would be worth, the revision adds expert-only distillation, i.e., BC trained only on the converged rollouts.

In the new Figure 10, BC on converged rollouts alone scores 0.99 on stairs up, and SOFIE with β≤1\beta\le1 scores 0.99 to 1.00. Once half the dataset consists of undertrained rollouts, SOFIE retains 0.90 without quality labels, while BC falls to 0.08. For IQL on expert data, the submitted Table 17 already provides the requested comparison: on flat ground, BC, IQL and SOFIE score 0.96, 0.95 and 0.94 with converged rollouts alone, compared with 0.25, 0.78 and 0.86 when half the data is undertrained.

DAgger, GAIL and inverse RL rely on resources that this setting excludes. DAgger queries the experts at the states visited by the student, so it needs the experts and the simulator during distillation (Ross et al., 2011). GAIL and inverse RL assume that all demonstrations are expert, and GAIL learns through environment interaction (Ho & Ermon, 2016; Ng & Russell, 2000). SOFIE needs no expert queries, no interaction and no label of which rollouts are expert. Within this setting, the matching teacher-student reference is the expert-only BC above, and SOFIE comes within a tenth of it without the label. Where experts and a simulator are available, these methods complement SOFIE, whose policy is a strong starting point for further online training (Figure 6a).

AuthorsOpenReview ↗

[post 2/2] Transfer and gait (W2), and scope (W3)

W2

"I cannot verify the robot's performance that claim to achieve perceptive locomotion without deploying on hardware. [...]"

The revision takes G1 beyond the training simulator in two new sim-to-sim tests whose observations are set close to those on the robot. In the first (Appendix E.4), a MuJoCo simulator runs the robot's deployment program with motor dynamics and noisy proprioception, and terrain observations come from elevation maps built from simulated range scans or noisy, delayed depth images. After BC fine-tuning on new G1 expert rollouts, the policies complete 28 or 29 of 30 stair flights and cover distances comparable to the experts (Table 16). The second (Appendix E.16) uses Unitree's deployment controller to test sensitivity to dataset quality, and SOFIE completes 21 of 24 stair climbs at 10% undertrained data and BC completes 1 (Figure 13). Hardware experiments on G1 are in preparation, and we will deploy the policies once the setup is complete.

"[...] M3 on stepping stones does not even look close to be deployable. The gait on g1 seems to be highly static on stepping stone and on downstairs it does not adjust the gait at all [...]"

We would like to ask the reviewer what they mean by "deployable". If it refers to a stable and robust gait, two new videos show M3 and G1 each crossing stepping stones, walking down and up a staircase and continuing over rough ground in one run without a fall. Each robot uses one 5M SOFIE policy fine-tuned online with the sparse reward. SOFIE provides the generalist pretrained policy, and online fine-tuning improves it further, from 0.76 to 0.97 on stairs up (new, Table 27).

The gait also adapts to the terrain. On the stepping stones, both robots keep walking and place their feet on the stones seen in the height scan, shown as dots in the videos. On stairs down, the policies depend on the height scan. In an ablation of the perceptive module, the policy receives the average of all training scans at every step (rather than zeros, to stay within the range seen in training), and the fine-tuned policies drop from 1.04 to 0.44 on stairs down (G1 alone from 0.99 to 0.71) and from 0.96 to 0.08 on rough ground (new, Table 28, Appendix E.13). Step timing also changes, as M3's median footstep lasts 17 control steps on rough ground and 30 to 31 on stairs and stepping stones (Figure 4b). The gait style comes from the specialists, whose rewards SOFIE does not change.

W3

"[...] it requires vast amount of data from an expert, it also requires the reward/advantage it experienced, it then assume the task must be offline (which I don't get because there's already an simulator). The method itself is very specific to the problem setting (bipedal locomotion)."

We believe that SOFIE is a step towards a foundation policy for perceptive humanoid locomotion: one policy pretrained on many robot-terrain pairs, from which later training can start instead of from scratch. This goal is why the setting is offline. The simulator trains each specialist, and SOFIE turns their recorded rollouts into one shared policy without further interaction. Each specialist was trained with a dense reward of 29 to 36 terms weighted for its robot-terrain pair (Appendix B.4), whereas fine-tuning from the foundation needs only a sparse reward that checks every 2 s whether the robot moved as commanded and penalizes falls. With this sparse reward on stairs up, training from scratch does not learn, while the SOFIE foundation reaches 0.83 (new, Figure 6a), and with the experts' dense reward on H1 stairs up, it reaches 0.92 using about a third of the environment steps that training from scratch needs (new, Figure 12).

As for data and rewards, SOFIE uses only what specialist training records anyway. The new Table 23 repeats the comparison of SOFIE and BC with one eighth of the data (128 rollouts per robot-terrain pair) and with Gaussian noise on the recorded rewards. SOFIE still outperforms BC with this smaller dataset, and with noise as large as the reward's standard deviation it keeps more than twice the score of BC on both terrains.

We chose perceptive humanoid locomotion as the testbed because it is well defined yet hard: one policy must serve several embodiments on challenging terrain, and the quality of an action often shows only at touchdown. The method itself is not specific to locomotion. It defines a segment (Section 4.1) as the trajectory between two events of a generic event detector, with footsteps as the instance used here, and the advantage weighting and quantile normalization apply to any such segments. Whole-body control, where motion waypoints or changes in contact could mark the segments, is left for future work. Within locomotion, the new fixed-window analysis shows that the weighting needs only an interval spanning at least one footstep, so it does not depend on contact detection (Table 22).

AuthorsOpenReview ↗

References

References

Ho & Ermon (2016). Generative Adversarial Imitation Learning. NeurIPS.

Ng & Russell (2000). Algorithms for Inverse Reinforcement Learning. ICML.

Ross et al. (2011). A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. AISTATS.

Reviewer_uaFKOpenReview ↗

Thank you for the response

I thank the authors for the response. Although the authors argue that the comparison is fair w.r.t baseline methods, I do not see the experiments themselves are valid justification or clear evidence such approach can work on real robots. I do not believe a sim2sim transfer is sufficient evidence. This work might have some real values, but I am simply not convinced. I find it perplexing on how the authors position this paper - if it is a method paper, please discuss its methodological improvement and address the fundamental issue. If it is a robotics paper, please add real robots, for otherwise there's almost no value to discuss perceptive locomotion, a mature field at 2026. Furthermore, claims seem to be made lightly, such as cross embodiment capability with limited robot types, etc. The assumption that the training data is not expert level and that being equivalent to half expert half middle training generated sample and use such mixed data for BC as baseline...I find it all to be carelessly placed. For these reasons, I keep my assessment for now.

Comments on the paper

2 comments
AuthorsOpenReview ↗

[post 1/2] General response

We thank the reviewers for their careful reading. The revision clarifies the learning setting and adds experiments that isolate the contributions of segment-level advantage weighting, quantile normalization and pretraining. Substantive changes are marked in blue in the revised PDF, to which all references below refer.

1. Learning one policy from mixed-quality rollouts (uaFK, oHEW)

Online RL trains locomotion specialists one robot and one terrain at a time, and perceptive locomotion on challenging terrain remains hard to train, even for one robot on one terrain. Each terrain skill needs its own curriculum, reward tuning and training procedure (Rudin et al., 2026, Sec. 2.1), and RPL trains its G1 terrain experts for 24 hours on four GPUs before distilling them for another 12 hours on eight (Zhang et al., 2026, Sec. IV-A). Even converged specialists remain imperfect: ours fall in 2.9 to 10.6% of episodes on stepping stones and rough ground at the hardest level (Table 5). A single policy across terrains is harder still, because RL on several terrains tends to focus on a subset of them and never find a solution for the others (Rudin et al., 2026, Sec. 4.1). Recent work therefore distills specialists, and we ask how their recorded rollouts can be turned into one shared perceptive policy. SOFIE learns the shared policy entirely offline from rollouts of both converged and earlier, undertrained checkpoints, without further environment interaction, access to the experts or labels identifying the quality of each rollout.

Such mixed-quality datasets contain useful and poor behavior side by side. BC gives every action equal weight and can therefore average conflicting behaviors. SOFIE instead estimates a footstep advantage from the recorded rewards and a value function fitted to the dataset, and it assigns the same weight to all actions within that footstep, so the actions that jointly place a foot are evaluated together. Quantile normalization within each robot-terrain group then keeps the weights comparable despite differences in reward scale.

Each setting trains all methods on the same dataset with the same policy architecture (Experiments, Section 5), and scores are normalized per robot-terrain pair so that its undertrained checkpoint scores 0 and its converged checkpoint 1. For one policy shared across three humanoids (G1, H1 and M3) and four terrains, SOFIE scores 0.68, compared with 0.28 for IQL and 0.18 for BC, while falling about half as often (Figure 11, Table 9). A new policy with a larger actor (5M instead of 1M parameters) reaches 0.92 (Figure 1).

2. Comparison with expert-only distillation (uaFK, oHEW)

Expert-only distillation trains BC on the rollouts of the converged checkpoints alone, which requires knowing which rollouts these are. We use it as the reference for how much performance SOFIE keeps without that knowledge. In a new experiment on stairs up with G1, H1 and M3 (Figure 10), we keep every converged rollout and add undertrained rollouts so that they make up between 0 and 50% of the dataset. On converged rollouts alone, BC scores 0.99 and SOFIE with mild weighting (β≤1\beta\le1) 0.99 to 1.00. When undertrained rollouts make up half the dataset, SOFIE retains 0.90 while BC drops to 0.08, so SOFIE approaches expert-only distillation without selecting the expert data in advance. A new sim-to-sim test with Unitree's deployment controller repeats this study for G1 on stairs up, and BC fails even earlier there: at 10% undertrained data, SOFIE completes 21 of 24 stair climbs and BC completes 1 (Figure 13).

AuthorsOpenReview ↗

[post 2/2] General response

3. Fine-tuning the pretrained policies (PCcH, oHEW)

We fine-tune the pretrained policies in two settings that answer different questions. In setting (i), supervised fine-tuning on new robots, policies pretrained with SOFIE, BC and IQL on the same source data (stairs up with G1, H1 and M3) are fine-tuned with BC on clean expert rollouts of three robots that are not in the source data (H3-1, T1 and T1VH). This setting measures the benefit of pretraining. In setting (ii), online RL on the training robots, the foundation policies, each pretrained with one method on G1, H1 and M3 across four terrains, are fine-tuned with REPPO and a sparse reward on stairs up. Training a specialist from scratch needs a dense reward with 29 to 36 terms weighted for each robot-terrain pair (Appendix B.4; Rudin et al., 2026, Sec. 2.1), whereas fine-tuning from the foundation needs only a sparse reward that checks every 2 s whether the robot moved as commanded and penalizes falls. This setting also measures whether the pretraining objective still matters when the policy must improve from its own behavior.

For setting (i), Table 15 now includes the BC and IQL initializations. All pretrained policies substantially outperform training from scratch, and SOFIE and BC reach similar returns, with IQL slightly lower, as expected when every initialization is fine-tuned on the same clean expert data. Setting (ii), which is new in the revision, separates the pretraining objectives. Before fine-tuning, SOFIE scores 0.71 on stairs up, compared with 0.28 for IQL and 0.15 for BC, and it ends highest at 0.83, while BC and IQL reach 0.57 and 0.56 and training from scratch with the same sparse reward reaches −0.11-0.11 (Figure 6a). The stronger SOFIE initialization retains its advantage during online learning.

4. Evaluation beyond the training conditions (uaFK, oHEW)

The main evaluations use the terrain level and the observations of the recorded datasets, so new experiments test whether the conclusions hold beyond them, on harder terrain (level 9 instead of 6), under random pushes and with observation noise (Table 24). SOFIE has the highest mean score under each shift, reaching 0.78 at terrain level 9 compared with 0.52 for IQL and 0.12 for BC. In another new experiment, we record every dataset again with a depth camera instead of the height scan and train every policy from these images. SOFIE then scores 0.87 and BC 0.49, with SOFIE leading on all twelve robot-terrain pairs with either sensor (Figure 1). Finally, two new sim-to-sim experiments take G1 outside the training simulator, once in a MuJoCo simulator that runs the robot's deployment program with noisy, delayed terrain observations (Table 16) and once with Unitree's deployment controller (Figure 13).

Main revisions (in blue)

  • The project page adds two videos of M3 and G1 crossing all terrains in one run.
  • Figure 1 adds the larger actor and depth camera. The submitted Figure 1 moves to Figure 11.
  • Figure 6a compares online fine-tuning from each pretrained policy on the training robots. The original panel is retained in Table 15.
  • The Results (Section 6) add expert-only BC and clarify that weighting intervals should span at least one footstep.
  • Appendix E adds sim-to-sim tests (E.4, E.16), weighting ablations (E.7, E.8), checkpoint selection and reward noise (E.9), evaluation beyond training conditions (E.10), model size and depth camera (E.12), online fine-tuning (E.13, E.14), and a second policy architecture (E.15).

References

  • Rudin et al. (2026). Parkour in the Wild: Learning a General and Extensible Agile Locomotion Policy Using Multi-Expert Distillation and RL Fine-Tuning. IJRR.
  • Zhang et al. (2026). RPL: Learning Robust Humanoid Perceptive Locomotion on Challenging Terrains. arXiv:2602.03002.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.