- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper come up with SOFIEm offline learning method for humanoid robots terrain traversal, which estimates segment advantages to truncated Monte Carlo returns, and train a morphology-conditioned policy. In the experiments, authors evaluate multiple humanoids on different terrains, which shows SOFIE improves performance than BC and offline RL.
Critical strengths
1.The segment-weighting idea is well motivated for locomotion
Critical weaknesses
-
The motivation is not grounded. The humanoid locomotion under perfect height map is been solved well in humanoid domain, and locomotion is so matured with PPO. I can train a policy like this with mjlab default setting + PPO within 3 hours even single gpu. Also the offline dataset with reward, state, contact label is easy to build in simulation but extremely tough to get in real world since you need mocap system, and feet contact estimation are always noisy. Also the dataset expert are actually from PPO, there are already a lot of papers that could one model control multiple robots like Locoformer.
-
The experiments are not convincing. Evaluation is same in distribution terrain level with training. Lack strong offline RL benchmark and baseline like decision transformer,ReBRAC...... Yet there is no wall-clock or GPU-hour comparison against online training or an expert-distillation pipeline. These make it unclear when the proposed pipeline offers a meaningful advantage beyond its fixed-dataset evaluation.
Questions for authors
Without an expert-only BC baseline in the perceptive experiments, what supports the claim that selectively learning from mixed-quality data is preferable to simply distilling the converged experts already available?
Since Table 15 compares SOFIE pretraining only against random initialization, what evidence attributes the transfer gains to SOFIE rather than to pretraining itself?
Given that the pipeline requires online-trained experts and substantial offline optimization, what's the advantage over online method?
Could you eval your method in a more standard and solid benchmark like OG bench? they also have env like humanoid and ant. And compare with SOTA offline RL/BC/goal condition BC policy
In general I think both motivation and experiments still need large improvement, I am really concerned that authors can't resolve my concern during rebuttal.
6 comments
[post 2/2] Evaluation (W2) and questions
W2, evaluation beyond training conditions
Because the main evaluations match the training conditions, the new Table 24 tests whether the performance differences persist when those conditions change. We evaluate the per-terrain policies on harder terrain, under random pushes and with observation noise, and normalize the scores under each condition with the corresponding expert and undertrained checkpoints.
| condition | SOFIE | IQL | BC |
|---|---|---|---|
| training conditions | 0.88 | 0.58 | 0.20 |
| terrain level 9 (three terrains) | 0.78 | 0.52 | 0.12 |
| random pushes | 0.72 | 0.53 | 0.19 |
| observation noise | 0.36 | 0.17 | 0.18 |
SOFIE has the highest mean score under each shift. Observation noise causes the largest drop for SOFIE and IQL, consistent with its absence from the recorded rollouts.
W2, baselines
The baselines cover BC and the offline RL methods relevant to these datasets, namely IQL, goal-conditioned IQL and TD3+BC (Experiments, Section 5). IQL is also the method Abe et al. (2026) use for cross-embodiment pretraining. The revision adds IQL with matched ranking groups and a second policy architecture, so that the effects of weighting and architecture can be examined separately (Tables 20 and 29). We did not evaluate Decision Transformer (Chen et al., 2021) or ReBRAC (Tarasov et al., 2023) and focused the additional experiments on controlled ablations of footstep weighting and quantile normalization (Tables 19 to 22).
W2, wall-clock cost
For the offline stage, SOFIE and BC use the same dataset and actor architecture, and SOFIE additionally fits a value function. Table 7 specifies 100k value-pretraining updates followed by 500k actor updates for the foundation policy, without further environment interaction. The new Figure 12 then measures the cost of subsequent online learning by comparing fine-tuning with training from scratch under the same dense reward. On H1 stairs up, the SOFIE initialization reaches 0.92 after 20M environment steps, while training from scratch reaches that score after 59M steps.
Q1, expert-only BC
Expert-only BC is the reference when converged data can be selected in advance, and the new Figure 10 measures how much of its performance survives when undertrained rollouts are added without quality labels. Expert-only BC scores 0.99, while SOFIE retains 0.90 when half the dataset is undertrained and BC on the same mixed dataset drops to 0.08 (point 2 of the General Response).
Q2, SOFIE or pretraining
We fine-tune the pretrained policies in two settings. With supervised fine-tuning on clean expert rollouts of three new robots (H3-1, T1 and T1VH), SOFIE and BC reach similar returns and IQL slightly lower ones (new rows of Table 15), since all three imitate the same expert data. With new online fine-tuning on the training robots (REPPO, sparse reward, stairs up), learning depends on the policy's own behavior, and the SOFIE foundation, which already starts best, ends highest at 0.83, compared with 0.57 for BC and 0.56 for IQL (Figure 6a).
Q3, comparison with online methods
Online RL trains the specialists, and SOFIE combines their skills offline without further interaction. New experiments then fine-tune the shared SOFIE policy online and compare it with training from scratch under the same reward. The foundation also changes what online RL needs: each specialist was trained with a dense reward of 29 to 36 terms weighted for its robot-terrain pair (Appendix B.4), whereas fine-tuning from the foundation works with a sparse reward that only checks every 2 s whether the robot moved as commanded and penalizes falls. With the experts' dense reward on H1 stairs up, the SOFIE initialization reaches 0.92 using about a third of the environment steps required from scratch (Figure 12), and with a sparse reward on stairs up, training from scratch does not learn while the SOFIE initialization reaches 0.83 (Figure 6a).
Q4, OGBench
Our experiments test footstep weighting and normalization within robot-terrain groups in shared perceptive policies. The humanoidmaze, antmaze and antsoccer tasks of OGBench (Park et al., 2025) study goal-conditioned offline RL with a single embodiment on flat ground, so they do not directly test this combination of policy sharing, challenging terrain and mixed-quality specialist rollouts. We therefore used the additional experiments to isolate the effects of segment weighting, ranking groups and dataset composition (Tables 19 to 23).
We believe these results address each of the reviewer's concerns, and we respectfully ask the reviewer to reconsider the rating in light of them.
References
References
Abe et al. (2026). Cross-Embodiment Offline Reinforcement Learning for Heterogeneous Robot Datasets. ICLR.
Ai et al. (2025). Towards Embodiment Scaling Laws in Robot Locomotion. CoRL.
Bohlinger et al. (2024). One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion. CoRL.
Chen et al. (2021). Decision Transformer: Reinforcement Learning via Sequence Modeling. NeurIPS.
Liu et al. (2025). LocoFormer: Generalist Locomotion via Long-context Adaptation. CoRL.
Park et al. (2025). OGBench: Benchmarking Offline Goal-Conditioned RL. ICLR.
Rudin et al. (2026). Parkour in the Wild: Learning a General and Extensible Agile Locomotion Policy Using Multi-Expert Distillation and RL Fine-Tuning. IJRR.
Tarasov et al. (2023). Revisiting the Minimalist Approach to Offline Reinforcement Learning. NeurIPS.
Zakka et al. (2026). mjlab: A Lightweight Framework for GPU-Accelerated Robot Learning. arXiv:2601.22074.
Zhang et al. (2026). RPL: Learning Robust Humanoid Perceptive Locomotion on Challenging Terrains. arXiv:2602.03002.
[post 1/2] Learning setting and observations (W1)
We thank the reviewer for finding segment-level advantage weighting well motivated. However, we respectfully disagree with the assessment that "the motivation is not grounded" and that "the experiments are not convincing." Several concerns treat the task as training an individual locomotion specialist, while the paper learns one policy shared across robots and terrains from recorded rollouts. We address each point below.
The mjlab default example (Zakka et al., 2025) trains G1 to follow velocity commands on flat ground, and we likewise use online RL to train each specialist for one robot-terrain pair (Appendix B.5). Our question begins after that step: how the recorded rollouts of these specialists can be used to learn one perceptive policy across three humanoids and four terrains. Combining the skills is hard, because RL on several terrains tends to focus on a subset of them and never find a solution for the others (Rudin et al., 2026, Sec. 4.1), which is why recent approaches distill terrain specialists. Even this route is costly for a single robot: RPL trains its perceptive G1 terrain experts for 24 hours on four GPUs and distills them for another 12 hours on eight GPUs (Zhang et al., 2026, Sec. IV-A).
Segment-level weighting needs temporal boundaries, which our tasks provide through the gait clock (Practical implementation, Section 4.2). These boundaries require neither contact labels nor motion capture, and the new fixed-window sensitivity analysis supports using any interval that spans at least one footstep (Table 22). The observations, actions and rewards are recorded in simulation during specialist training, so the learning setting does not depend on recovering these quantities from real-world logs.
To clarify this, we refer the reviewer to the Introduction and the Related Work (Section 2). We note that none of the multi-robot policies discussed there perceives the terrain. URMA omits exteroceptive sensors (Bohlinger et al., 2024, Sec. 3), GenBot-1K is proprioceptive and studies flat terrain only (Ai et al., 2025), and LocoFormer acts on observations in a unified joint space, without a height map, depth or camera (Liu et al., 2025, Sec. 2.1). Perception matters on our terrains. To evaluate the importance of the perceptive module, we replace the height scan at test time with the average of all training scans (rather than zeros, to keep the input within the range seen in training). Our fine-tuned policies then lose more than half of their score on stairs down and almost all of it on rough ground (new, Table 28). In contrast, SOFIE learns one perceptive policy that controls several humanoids on stairs and stepping stones, entirely from recorded rollouts of mixed quality. We additionally evaluate BC following the recipe of GenBot-1K (Experiments, Section 5) and SOFIE with URMA's network (new, Table 29).
We respectfully disagree. To the best of our knowledge, one perceptive policy that controls several humanoid embodiments across challenging terrains has not been learned before. The Related Work (Section 2) reviews the two closest lines of work, perceptive policies across terrains for a single robot (Rudin et al., 2026; Zhang et al., 2026) and policies shared across robot designs (Bohlinger et al., 2024; Ai et al., 2025). Even single-robot specialists on one terrain remain imperfect: ours still fall in 2.9 to 10.6% of episodes on stepping stones and rough ground at the hardest level (Table 5), and RL on several terrains tends to cover only a subset of them (Rudin et al., 2026, Sec. 4.1). A new experiment also shows that SOFIE does not depend on a height map. With every dataset recorded again from a depth camera, SOFIE scores 0.87 compared with 0.49 for BC and leads on every robot-terrain pair with either sensor (Figure 1). The new sim-to-sim experiments go further and set the observations close to those on the robot, using an elevation map built from simulated range scans under drifting odometry or noisy, delayed depth images. After fine-tuning, the policies complete 28 or 29 of 30 stair flights (Table 16).