- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors consider the problem of evaluating neural network architectures for robot control policies. They focus specifically on the case where a single student policy is trained to imitate a bank of expert policies, each trained for a different physics setting. They present a study that shows that changes in training data and evaluation conditions often affect the evaluation performance, and, therefore, the architecture rankings. Their contributions, as I understand them, are a set of statistical analyses showing that training data and evaluation conditions change the ranking, an explanation for this phenomenon, and a proposed reporting protocol that accounts for this variability.
Critical strengths
The motivation is clear, and the statistical analyses, as far as I can tell, are sound. Their findings seem to support that evaluation conditions and expert labels affect architecture performance (for their particular experimental setup). For example, Figure 5 clearly shows how simply varying evaluation conditions can flip the relative ranking of two architectures. These results are important because evaluators who are not aware of this relationship may come to the wrong conclusions about which architecture does best.
Critical weaknesses
It is very hard to make sense of the writing. Here is one example from contribution 3 on the first page: "Reversals across test sets follow from how much a gap varies across physics and can be predicted from separate data. Inside a single policy, the expert decides where accurate physics hurts, and relabeling alone repairs or creates these failures."
- What does it mean for a "gap" to "vary across physics"?
- What is this "separate data"?
- Why is physics "accuracy" being mentioned at all?
- What is relabeling? None of these terms are defined or clear at this point in the paper.
Reading the corresponding results in Section 4.3 doesn't help either. For example, the authors start by referencing Equation 5, which was first defined 4 pages earlier and not used until this moment. Moreover, the language is not clear at all: "Equation 5 explains why a precise result on one test set can mislead: a new test set also redraws the physics. Adding the physics term to the same 192 comparisons keeps every estimate but cuts supported rankings from 100 to 9 (Appendix I.2).
We turn this into a risk score.[...]"
- It is not clear to me why Equation 5 explains anything at all.
- What is meant by "adding" a "physics term"?
- What comparisons?
- What is being turned into a risk score, and why does this make sense?
The paper is full of similarly vague, undefined, or disconnected statements.
Even if the writing were clear, the significance of this work is unclear to me.
Their study seems to be on a very specific evaluation setup, where a single student policy is trained to match a bank of experts trained using PPO (which, by the way, are not quite "experts" in the conventional sense of the word because many of them don't perform very well on the tasks they purport to be experts at, cf. Figure 3).
The paper does not establish that this evaluation setup is common and/or relevant in the literature, which makes me skeptical that it deserves such a thorough analysis.
Moreover, there are no references after 2023, which is not an issue by itself, but it does not inspire confidence that this study would transfer to modern policy evaluation setups.
Questions for authors
The related work covers distillation from many experts and physics-conditioned policies, but I don't see any references to prior work that use the setup you study, i.e., per-setting experts distilled into a single physics-conditioned student. Can you point to prior work that trains or evaluates policies this way and/or anything to suggest your results would transfer to more common setups (e.g., RL policies trained with domain randomization)? This would help strengthen the importance/relevance of your work.
2 comments
I appreciate the authors response and additional context.
However, the following concerns remain:
- The writing in the original paper was confusing enough that I would not feel comfortable recommending this paper for acceptance unless I were to see clear changes already with an updated PDF. The exposition issues I pointed out were major, and saying that they will be fixed is not enough.
- In fact, given one of my initial concerns, they propose renaming "experts" to "teacher". And then, in the next paragraph they start calling them "specialists." In my experience, use of jargon in this manner is a tell-tale sign of AI-generated text. So I ran this through Pangram and it came up 94% AI-generated, which does not inspire confidence at all.
- The relevance of their setup is still unclear. In practice, if a "teacher" or "expert" policy was not good at the task it was trained on, it would simply not be used as a teacher policy!
I maintain my score.
Thank you for the detailed review. We agree with your points on the writing and are revising the paper accordingly.
1. Writing. We agree: we used shorthand terms ("gap", "physics term", "relabeling") without defining them. We will replace shorthand with plain words, define each term where it first appears, and use one term per concept throughout.
By "gap" we meant the difference in mean return between two policies. It "varies across physics" when it changes between physics settings. "Separate data" means 256 physics settings that no test set uses. "Accurate physics" means the true physics values instead of the default ones. "Relabeling" means changing the target actions of some training settings, keeping their states.
Eq. 5 splits the uncertainty of this difference into two parts: which physics settings are in the test set, and the random start of each episode. A usual interval fixes the test set and covers only the second part, but another test set could give a different result. "Adding the physics term" means including the first part. The comparisons are the 192 of Section 4 (24 policy pairs, each on eight test sets).
The risk score is the chance that, among eight new test sets, some name one policy the clear winner and others the other. We estimate the difference and both variance parts on the 256 separate settings, compute the chance of a clear winner in each direction for one test set, and combine these for eight sets (Eq. 6). We will move Eq. 5 to Section 4.3, where it is used.
2. "Experts." Agreed; we will call them teacher policies. That the policy trained at a setting is often not the best teacher for it is part of our point: it makes the choice of teacher a real decision, and that decision changes which architecture looks best.
3. Prior work with this setup. Yes. Distilling per-condition RL specialists into one student is common, especially in locomotion: Robot Parkour Learning (CoRL 2023) distills five RL skill specialists into one vision-based policy, Barkour (2023) distills per-obstacle specialists into one Locomotion-Transformer, and Divide-and-Conquer RL (ICLR 2018) distills per-context local policies into one global policy.
In these pipelines, each specialist labels its own condition by design, and these papers do not test this choice. But the specialist trained at a condition is often not the best teacher there: in our banks, another teacher scores higher at 33 of 44 training settings. We study what this hidden choice does. Changing only which teacher labels each condition changes which architecture looks best, and it determines where accurate physics inputs hurt. We will add this motivation to the introduction.
4. Transfer to RL with domain randomization. Yes, there is evidence. We trained PPO with domain randomization on the same 11 settings (3M steps, the teachers' hyperparameters, three seeds), with the physics input (θ) and without it (blind), and compared them as in Section 4. First, a confident conclusion from one training run can reverse in another: on 256 separate settings, θ minus blind is +357 in one Walker2d run and −414 in another, both with intervals far from zero; in HalfCheetah, it is +740 in one run and no effect (−2) in another. Second, the true physics input can hurt: in Ant, it lowers return in all three seeds. Third, the physics variance matters: in Walker2d, including it cuts the comparisons with a clear winner from 19 to 11 of 24. These are the effects we report for BC students.
5. Recent references. Our core message is that conclusions about physics-conditioned policies depend on two rarely reported choices, the teacher that labels each setting and the physics used for testing, and that both can be anticipated in advance: the action gap between neighboring teachers predicts where physics inputs hurt (AUC 0.89), and a risk score from separate data predicts which comparisons reverse (AUC 0.81).
Recent work makes both choices without testing them. Distillation pipelines for locomotion fix each expert to its own condition, for example Rudin et al. (2025), "Parkour in the Wild: Learning a General and Extensible Agile Locomotion Policy Using Multi-expert Distillation and RL Fine-tuning", and Wang et al. (2026), "X-Loco: Towards Generalist Humanoid Locomotion Control via Synergetic Policy Distillation". Iannotta et al. (2025), "Can Context Bridge the Reality Gap? Sim-to-Real Transfer of Context-Aware Policies", find that context helps but that the best way to use it depends on the task. None of the three tests whether its conclusion survives another teacher assignment or test set.
Recent guidance does not cover it either: neither the survey by Mohan et al. (2024), "Structure in Deep Reinforcement Learning: A Survey and Open Problems", nor the guidelines by Patterson et al. (2024), "Empirical Design in Reinforcement Learning", discusses the choice of teacher. We will add this to the introduction.