acceptodds
Under review as a conference paper at ICLR 2027

Conditional Policy Advantage: Changing the Expert or Test Physics Can Reverse Policy Rankings

Abstract

Does telling a robot controller the physics it faces, such as gravity, really help it handle changing physics? Across four simulated locomotion tasks, we show that the answer, and which architecture uses physics best, depends on two rarely reported choices: the expert whose actions the controller learns to copy, and the physical conditions used for testing. With training states fixed, changing only the expert reverses architecture rankings. With trained controllers fixed, changing only the test physics reverses whether physics inputs help. Even test sets drawn from one distribution rank architectures in opposite orders in 7 of 24 comparisons. These reversals are predictable from separate data, and inside a single controller the expert decides where accurate physics hurts. We call this dependence Conditional Policy Advantage and propose an Expert-by-Test Grid to report it.

Primary area:
reinforcement learning
Keywords:
imitation learning, physics-conditioned policies, teacher selection, policy evaluation, generalization

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 5 reviews, above 0% of the others.
Reviewer_NYs9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors consider the problem of evaluating neural network architectures for robot control policies. They focus specifically on the case where a single student policy is trained to imitate a bank of expert policies, each trained for a different physics setting. They present a study that shows that changes in training data and evaluation conditions often affect the evaluation performance, and, therefore, the architecture rankings. Their contributions, as I understand them, are a set of statistical analyses showing that training data and evaluation conditions change the ranking, an explanation for this phenomenon, and a proposed reporting protocol that accounts for this variability.

Critical strengths

The motivation is clear, and the statistical analyses, as far as I can tell, are sound. Their findings seem to support that evaluation conditions and expert labels affect architecture performance (for their particular experimental setup). For example, Figure 5 clearly shows how simply varying evaluation conditions can flip the relative ranking of two architectures. These results are important because evaluators who are not aware of this relationship may come to the wrong conclusions about which architecture does best.

Critical weaknesses

It is very hard to make sense of the writing. Here is one example from contribution 3 on the first page: "Reversals across test sets follow from how much a gap varies across physics and can be predicted from separate data. Inside a single policy, the expert decides where accurate physics hurts, and relabeling alone repairs or creates these failures."

  • What does it mean for a "gap" to "vary across physics"?
  • What is this "separate data"?
  • Why is physics "accuracy" being mentioned at all?
  • What is relabeling? None of these terms are defined or clear at this point in the paper.

Reading the corresponding results in Section 4.3 doesn't help either. For example, the authors start by referencing Equation 5, which was first defined 4 pages earlier and not used until this moment. Moreover, the language is not clear at all: "Equation 5 explains why a precise result on one test set can mislead: a new test set also redraws the physics. Adding the physics term to the same 192 comparisons keeps every estimate but cuts supported rankings from 100 to 9 (Appendix I.2).

We turn this into a risk score.[...]"

  • It is not clear to me why Equation 5 explains anything at all.
  • What is meant by "adding" a "physics term"?
  • What comparisons?
  • What is being turned into a risk score, and why does this make sense?

The paper is full of similarly vague, undefined, or disconnected statements.

Even if the writing were clear, the significance of this work is unclear to me. Their study seems to be on a very specific evaluation setup, where a single student policy is trained to match a bank of experts trained using PPO (which, by the way, are not quite "experts" in the conventional sense of the word because many of them don't perform very well on the tasks they purport to be experts at, cf. Figure 3). The paper does not establish that this evaluation setup is common and/or relevant in the literature, which makes me skeptical that it deserves such a thorough analysis.
Moreover, there are no references after 2023, which is not an issue by itself, but it does not inspire confidence that this study would transfer to modern policy evaluation setups.

Questions for authors

The related work covers distillation from many experts and physics-conditioned policies, but I don't see any references to prior work that use the setup you study, i.e., per-setting experts distilled into a single physics-conditioned student. Can you point to prior work that trains or evaluates policies this way and/or anything to suggest your results would transfer to more common setups (e.g., RL policies trained with domain randomization)? This would help strengthen the importance/relevance of your work.

2 comments

Thank you for the detailed review. We agree with your points on the writing and are revising the paper accordingly.

1. Writing. We agree: we used shorthand terms ("gap", "physics term", "relabeling") without defining them. We will replace shorthand with plain words, define each term where it first appears, and use one term per concept throughout.

By "gap" we meant the difference in mean return between two policies. It "varies across physics" when it changes between physics settings. "Separate data" means 256 physics settings that no test set uses. "Accurate physics" means the true physics values instead of the default ones. "Relabeling" means changing the target actions of some training settings, keeping their states.

Eq. 5 splits the uncertainty of this difference into two parts: which physics settings are in the test set, and the random start of each episode. A usual interval fixes the test set and covers only the second part, but another test set could give a different result. "Adding the physics term" means including the first part. The comparisons are the 192 of Section 4 (24 policy pairs, each on eight test sets).

The risk score is the chance that, among eight new test sets, some name one policy the clear winner and others the other. We estimate the difference and both variance parts on the 256 separate settings, compute the chance of a clear winner in each direction for one test set, and combine these for eight sets (Eq. 6). We will move Eq. 5 to Section 4.3, where it is used.

2. "Experts." Agreed; we will call them teacher policies. That the policy trained at a setting is often not the best teacher for it is part of our point: it makes the choice of teacher a real decision, and that decision changes which architecture looks best.

3. Prior work with this setup. Yes. Distilling per-condition RL specialists into one student is common, especially in locomotion: Robot Parkour Learning (CoRL 2023) distills five RL skill specialists into one vision-based policy, Barkour (2023) distills per-obstacle specialists into one Locomotion-Transformer, and Divide-and-Conquer RL (ICLR 2018) distills per-context local policies into one global policy.

In these pipelines, each specialist labels its own condition by design, and these papers do not test this choice. But the specialist trained at a condition is often not the best teacher there: in our banks, another teacher scores higher at 33 of 44 training settings. We study what this hidden choice does. Changing only which teacher labels each condition changes which architecture looks best, and it determines where accurate physics inputs hurt. We will add this motivation to the introduction.

4. Transfer to RL with domain randomization. Yes, there is evidence. We trained PPO with domain randomization on the same 11 settings (3M steps, the teachers' hyperparameters, three seeds), with the physics input (θ) and without it (blind), and compared them as in Section 4. First, a confident conclusion from one training run can reverse in another: on 256 separate settings, θ minus blind is +357 in one Walker2d run and −414 in another, both with intervals far from zero; in HalfCheetah, it is +740 in one run and no effect (−2) in another. Second, the true physics input can hurt: in Ant, it lowers return in all three seeds. Third, the physics variance matters: in Walker2d, including it cuts the comparisons with a clear winner from 19 to 11 of 24. These are the effects we report for BC students.

5. Recent references. Our core message is that conclusions about physics-conditioned policies depend on two rarely reported choices, the teacher that labels each setting and the physics used for testing, and that both can be anticipated in advance: the action gap between neighboring teachers predicts where physics inputs hurt (AUC 0.89), and a risk score from separate data predicts which comparisons reverse (AUC 0.81).

Recent work makes both choices without testing them. Distillation pipelines for locomotion fix each expert to its own condition, for example Rudin et al. (2025), "Parkour in the Wild: Learning a General and Extensible Agile Locomotion Policy Using Multi-expert Distillation and RL Fine-tuning", and Wang et al. (2026), "X-Loco: Towards Generalist Humanoid Locomotion Control via Synergetic Policy Distillation". Iannotta et al. (2025), "Can Context Bridge the Reality Gap? Sim-to-Real Transfer of Context-Aware Policies", find that context helps but that the best way to use it depends on the task. None of the three tests whether its conclusion survives another teacher assignment or test set.

Recent guidance does not cover it either: neither the survey by Mohan et al. (2024), "Structure in Deep Reinforcement Learning: A Survey and Open Problems", nor the guidelines by Patterson et al. (2024), "Empirical Design in Reinforcement Learning", discusses the choice of teacher. We will add this to the introduction.

I appreciate the authors response and additional context.

However, the following concerns remain:

  1. The writing in the original paper was confusing enough that I would not feel comfortable recommending this paper for acceptance unless I were to see clear changes already with an updated PDF. The exposition issues I pointed out were major, and saying that they will be fixed is not enough.
  2. In fact, given one of my initial concerns, they propose renaming "experts" to "teacher". And then, in the next paragraph they start calling them "specialists." In my experience, use of jargon in this manner is a tell-tale sign of AI-generated text. So I ran this through Pangram and it came up 94% AI-generated, which does not inspire confidence at all.
  3. The relevance of their setup is still unclear. In practice, if a "teacher" or "expert" policy was not good at the task it was trained on, it would simply not be used as a teacher policy!

I maintain my score.

Reviewer_AickOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work is focused on training unified student policies to imitate pretrained experts across different physical conditions in simulated robot locomotion tasks. Its key finding is: architecture rankings depend on both which experts supply the training action labels and which physical conditions are used for evaluation. Holding training states and training procedures fixed, changing only the expert assignment and retraining the students can reverse architecture rankings. Holding trained students fixed, changing only the test physics can also reverse rankings, even across independently sampled test sets from the same distribution. The results show that an architecture's advantage under one teacher assignment and one test set need not generalize, motivating evaluation across multiple teacher assignments and test conditions.

Critical strengths

None

Critical weaknesses

I feel confused by the paper. The significance and generalizability of the findings remain unclear. While the controlled experiments demonstrate that architecture rankings can reverse under changes to expert assignments or test physics, the paper does not sufficiently establish why these reversals are consequential beyond the expected sensitivity of methods with modest performance differences to training and evaluation choices. It remains unclear which important existing conclusions these findings challenge or what actionable guidance they provide beyond broader evaluation and appropriate uncertainty estimation. I'm also not sure where the conclusions can be applied.

Questions for authors

I learned very little from the paper and would suggest the authors to either significantly improve writing to highlight why this is an important problem to work on, or start with an important problem before conducting any future research.

1 comment

Thank you for the review.

1. Why the reversals matter. They change which student design a study recommends. Take two common ways to give the student the physics values: Concat appends them to the input, and FiLM uses them to scale the hidden layers. In HalfCheetah (three seeds averaged), when each setting is taught by its own teacher, FiLM beats Concat (585 vs. 404). When each setting is taught by its best teacher, Concat beats FiLM (1034 vs. 957). The recommendation therefore depends on a choice that papers do not report. That choice also matters more than the design: changing the teacher moves returns by 370 to 630, changing the design by only 80 to 180. These are not near-ties flipping: pairs that reversed started with gaps as large as pairs that did not (median 231 vs. 213), and the shift holds over five seeds (+215, p = 0.007).

2. Which conclusions this challenges. Mainly an assumption that current distillation pipelines build on: that the specialist trained at a condition is the right teacher for it. Parkour in the Wild (Rudin et al., 2025) and X-Loco (Wang et al., 2026) both fix each expert to its own condition. This often fails. In our banks another teacher is better at 33 of 44 settings, and this hidden choice can decide which student design wins. Work on context-aware policies, such as Iannotta et al. (2025), likewise does not test whether its conclusion survives another teacher assignment or test set. Our work makes both choices checkable in advance, before any evaluation is run.

3. Actionable guidance beyond broader evaluation. The guidance works before evaluation, not after. Before training, compare the actions of teachers at neighboring settings. Large gaps predict where physics inputs will hurt (AUC 0.89), and relabeling only those settings removes the failures (12/12 intervals above zero), with no new architecture or environment data. Before testing, compute the risk score on separate data. It predicts which comparisons will reverse (AUC 0.81), so they are not reported as settled.

4. Where the conclusions apply, and what they enable. Anywhere a controller is built from several specialists or compared on sampled physics settings. In both cases, our checks move the decision from after the experiment to before it. A team learns which comparisons it can trust before spending its training and testing budget, instead of finding out after a result fails to hold.

Building a controller by distillation. A team trains one RL specialist per terrain or body variation and distills them into one controller, as in Parkour in the Wild (Rudin et al., 2025) and X-Loco for humanoids (Wang et al., 2026). Before training, our check shows where neighboring specialists disagree so much that giving the controller the physics values will backfire. Relabeling those conditions with another specialist fixes it. The team avoids a controller that fails at those conditions, using only the specialists it already has.

Comparing controllers in simulation. A team compares two designs on a few dozen sampled physics settings and reports the winner, however the controllers were trained. Before testing, our risk score, computed on separate settings, tells the team whether that winner is likely to flip on another sample. The team avoids reporting, or shipping, a winner that does not hold. We checked this on PPO policies trained with domain randomization, with no distillation. In Walker2d, for example, once the variance across physics is counted, the comparisons with a clear winner drop from 19 to 11 of 24.

5. What is new, and why it matters. Four results that current practice does not anticipate. The specialist trained at a condition is not its best teacher at 33 of 44 settings, and across our HalfCheetah banks the best one scores 169 to 856 higher on average. Changing only the teacher reverses 49 of 160 design comparisons. Standard intervals declare a clear winner in 100 of 192 comparisons, but counting the variance across physics leaves only 9. Giving a policy accurate physics values can hurt it, at settings that can be predicted before training (AUC 0.89). This matters because robot controllers are increasingly built by distilling RL specialists, from quadruped parkour to humanoids, and design comparisons decide what gets trained, published and deployed. Checking in advance saves cost: a team no longer spends training and testing runs on comparisons that will not hold. We will make this clearer by tightening the logic of the paper and the links between its sections.

Reviewer_kvdZOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how the choice of expert and test physics affects comparisons between physics-conditioned policies trained by behaviour cloning. At each training setting, the student imitates either the expert trained there (OWN) or the expert with the highest observed return there (BEST). Changing the expert labels on the same training states can change which architecture earns the higher return. With the trained policies fixed, one can perform better on one set of test physics and the other can perform better on another set. The authors propose an Expert-by-Test Grid to report how return differences vary with both choices.

Critical strengths

(S1. Interesting observation) I would expect a student’s performance to change with the test physics or the expert it imitates. What surprised me is that, at 33 of the 44 training settings, an expert trained elsewhere achieved a higher observed mean return than the specialist trained at that setting. This makes me wonder whether some PPO "specialists" settled on weaker policies during optimisation, and how much the gap depends on the training run.

(S2. A promising way to flag fragile rankings) Section 4.3 helps identify when a clear winner on one set of physics settings may lose on another drawn from the same distribution. I find that useful for deciding when to test new settings, even if repeated runs at the original settings agree, although the score requires substantial computation/evaluation.

Critical weaknesses

(W1. The headline claim is too broad) The abstract/intro says that “changing only the expert reverses architecture rankings,” or "changing only the test physics reverses whether physics inputs help", which makes reversal sound like a general outcome. However, in Table 3, with training states and test physics fixed, switching from OWN to BEST produced supported reversals in 49 of the 160 pairwise comparisons. Figure 2 also showed how much this reversal depends on the task.

(W2. Limited insight into what makes a better teacher) The observation in S1 raises a more interesting question for me. When an expert trained elsewhere outperforms the specialist on its own setting, what makes that expert’s actions more useful to imitate? On HalfCheetah, switching from OWN to BEST substantially improves Concat’s mean return while Gated-LoRA’s return barely changes (Table 2). The reporting protocol helps reveal this difference, but I would have liked a follow-up to explain why a higher-return teacher benefits one student much more than another. That would give the observation clearer implications for policy design.

(W3. Why does the choice of teacher favour a different architecture?) Follow up from W2, Figure 2 is the result I most wanted the paper to explain. The figure reports the difference between the students, but I cannot tell how much each student’s return changes or whether a few test settings account for most of that change. Section5/Figure 7 establishes that changing labels can cause failures for Concat, but the intervention tests Concat alone. I am therefore left with an explanation for particular failures in one student and an open question about the relative behaviour of the two architectures. This is the main limitation on the significance of the paper for me.

Questions for authors

Q is for questions and C is for comments.

(Q1. Addresses W1) What distinguishes the pairs that reversed from those that retained their ordering? For example, did reversals mostly occur when the initial return gap was small?

(Q2. Addresses W2) On HalfCheetah, why does switching to BEST substantially improve Concat’s return while Gated-LoRA’s return barely changes (Table 2)? At training settings where the teacher changes, does either/both student fit the new actions better? Does that difference track the new teacher’s return advantage or how much its actions differ from the OWN teacher’s on the same states?

(Q3. Addresses W3) For each of the three HalfCheetah fits in Figure 2, how does switching from OWN to BEST change FiLM’s and Concat’s individual returns, and which held-out physics settings contribute most to the reversal? Is the difference between the students already visible in their imitation errors on the same training states, or does it emerge mainly during rollouts?

(Q4. Does sharing a teacher help one architecture more than the other?) Section 3.3 points out that BEST can assign the same expert to several physics settings. When this happens, the new teacher would give the same action for a given state at both settings, even if the original teachers would have disagreed. Choosing a higher-return expert can therefore also make the imitation targets more consistent across settings. I would like to know whether Concat and FiLM benefit differently from that change. In the HalfCheetah runs from Figure 2, how does each student’s imitation error change at settings that come to share a teacher under BEST? Does either architecture benefit more from sharing a teacher, particularly where the original teachers prescribed very different actions on the same states?

(C1. Flow of the experiments) I found the purpose of individual experiments easier to understand, but not the connection between each experiments. A short transition explaining which question does each experiment answer would help readers follow the overall story and see where the paper returns to the original architecture comparison.

2 comments

Thank you for the careful reading. We address each point below.

S1 (Weak specialists? Run dependence?) Weak specialists are exactly why the teacher choice matters, and the gap is not a run artifact. In all four independently trained HalfCheetah banks, 7-9 of 11 settings have a teacher stronger than the specialist, and it stays ahead on fresh resets by 169-856 on average (Appendix C). At five settings, including both gravity extremes, the specialist loses in all four banks.

S2 (Cost of the risk score). Thank you. We will state the settings and episodes the score needs (256 separate settings x 4 resets per policy), so that readers can judge its cost.

W1/Q1 (How general are reversals? Were they close calls?) We will state rates in the abstract. 49 of 160 is a conservative count: we only count reversals where both intervals exclude zero. Among the 111 comparisons where a reversal can be confirmed (a clear winner before and after), 49 (44%) flip, across all four tasks and 22 different architecture pairs. The effect is not confined to one task or one pair. Reversed pairs were not close calls: the initial Own gap is the same for reversed and kept pairs (median 231 vs. 213; AUC 0.53).

W2/Q2 (What makes a better teacher? Why does Gated-LoRA not gain?) The Best teacher is a generally stronger policy, not only better at its own setting: in the original HalfCheetah bank, one teacher is best at 7 of 11 settings, with a mean return over all settings of 1397 vs. 782 for the median teacher. Under Best, most settings are therefore labeled by one strong teacher, and both students fit the new labels better at the 9 switched settings (median error x0.66 for Concat, x0.69 for Gated-LoRA).

Gated-LoRA does gain from Best, but only at the training settings. Rolled out there (10 resets per setting), both students gain in every seed (Concat +413 to +459, Gated-LoRA +289 to +437). On the held-out test settings, Concat gains +510 to +780 and Gated-LoRA -44 to +54. Under Own, Concat keeps only 24-40% of its training-setting return on new physics, Gated-LoRA 70-75%; Best raises Concat's share to 61-65%. Conflicting Own labels thus mainly damage Concat's transfer to new physics (the mechanism of Section 5), and consistent labels repair it. Gated-LoRA already transferred well, so Best adds nothing on the test settings.

W3/Q3 (Figure 2: individual returns, which settings, imitation error?)

(1) Individual returns. Both students improve, and Concat improves more in every fit, which flips the ranking.

HalfCheetah fitConcat Own→BestFiLM Own→BestSettings where Concat gains more
seed 42262→1065 (+803)548→982 (+434)11/13
seed 1456→1067 (+611)520→978 (+458)8/13
seed 2495→969 (+474)686→911 (+225)9/13

(2) Which settings. Five two-axis test settings that pair low mass or friction with high damping or armature. They favor Concat in every fit and carry the entire net effect. The largest is mass 0.75 with damping 2.0 (Concat's extra gain +1234 on average).

(3) Imitation error or rollouts? Mainly rollouts on new physics. Concat's imitation error is 24-26% higher than FiLM's under Own and 14% under Best, but this does not cost return at the training settings: rolled out there, the two are equally good under both assignments (Own 1195-1201 vs. 1184-1202; Best 1637-1667 vs. 1549-1643). The difference is how much of this return each keeps on new physics: under Own, Concat keeps 22-41% and FiLM 44-57%; under Best, both keep about 60%. Conflicting Own labels thus damage Concat's transfer more than FiLM's, and consistent labels repair it. Section 5 shows where these conflicts arise.

Q4 (Does sharing a teacher help one architecture more?) In HalfCheetah, Best replaces 11 teachers with 3, and all 9 switched settings come to share a teacher.

(1) Imitation error at these settings. Both students fit better: Concat's error falls by 34%, FiLM's by 28%.

(2) Which architecture benefits more? Concat, as you hypothesized, in every fit. The gap is largest where the original teachers disagreed most (44% vs. 36%) and small where they agreed (23% vs. 20%). We will add this analysis to Section 3.3.

C1. Thank you. We will add transitions between the experiments to make the paper easier to follow.

Reviewer_kvdZOpenReview ↗

reponses to rebuttal

I appreciate the authors for the additional analyses. They clarify that both students improve under BEST, that Concat improves more, and that particular held-out physics settings account for the reversal. The distinction between performance at training and test settings also addresses some parts of my questions.

My main concern, however, remains about what we learn beyond the observation that architecture rankings change with teacher and evaluation choices. The response explains more clearly where the reported differences occur, but provides limited insight into why the architectures respond differently or when can we expect a particular architecture to benefit. This was the central concern behind W2 and W3.

I therefore maintain my score.

Reviewer_auGDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies BC in a contextual MDP from experts that cover a sparse set of varying physical configurations. The authors consider variables related to hte physics of the environments, such as gravity, friction, mass, damping, and armature. Then, they train an expert for each config and do BC on these experts to study how the choice of expert and evaluation settings affect the resulting policies. They have 3 major findings: Changing test settings can reverse whether supplying physical parameters helps; providing the correct parameter sometimes reduces performance relative to providing the nominal context; and changing which expert supplies the training actions can create or reduce failures at configurations between those used for training, with relabeling experiments showing that expert choice can create or reduce failures at intermediate configurations. Based on these, they recommend evaluating across multiple experts and test sets.

Critical strengths

I think the study is potentially interesting to the question of how to do BC in cMDPs, what kind of limitations can exist in such settings, and how could one mitigate them

Critical weaknesses
  • The writing needs to be more precise overall. I think the paper uses awkward phrasing, making it difficult to follow. Some examples
    • “Giving the gravity and damping endpoints the nominal expert’s labels largely restores the halfway settings”
    • “Adding the physics term … keeps every estimate but cuts supported rankings”
    • “The expert decides where accurate physics hurts”
  • I do not think 3 seeds are enough for statistical significance. I recommend running at least 5
  • Literature: The paper misses some major related literature:
    • Contextual MDPs and Zero-shot generalization: The problem has been previously formalized in multiple places, but I think cMDPs might be the closest [1,2,3,4]. CARL [2, Sections 6.3–6.4] already shows that providing context can help in some settings and hurt in others for RL, and examines how generalization depends on the training distribution and evaluation region. The authors should discuss and place their work in this literature, and specifically refine what the work adds
    • Multi-modal BC and Domain randomization: I recommend also looking at this literature [6.7. 8,9] w.r.t. training coverage and generalization across physical conditions. Here, experts are trained with one parameter changed at a time, while testing combines changes across parameters.
  • Although the authors ablate the architecture, all BC learners learn using MSE. Given multiple experts, I am unclear about whether these failures could also arise from action averaging. I recommend trying a different BC objective, such as a multimodal BC model like a mixture of Gaussians
  • Coverage of parameter combinations: I recommend adding joint variations fo parameters to hte analysis (see Fig. 3 in [3])

I think the paper has merit and could become a useful empirical study. However, the current draft needs to position itself clearly and be more precise about what the experiments show. I believe the paper can be strengthened significantly with more grounding in the theory of contextual MDPs, and the experiments need ablations to support the claims.

[1] Hallak, A., Di Castro, D., & Mannor, S. (2015). Contextual markov decision processes. arXiv preprint arXiv:1502.02259.

[2] Benjamins, C., Eimer, T., Schubert, F., Mohan, A., Döhler, S., Biedenkapp, A., ... & Lindauer, M. (2022). Contextualize me--the case for context in reinforcement learning. arXiv preprint arXiv:2202.04500.

[3] Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023). A survey of zero-shot generalisation in deep reinforcement learning. Journal of Artificial Intelligence Research, 76, 201-264.

[4] Mohan, A., Zhang, A., & Lindauer, M. (2024). Structure in deep reinforcement learning: A survey and open problems. Journal of Artificial Intelligence Research, 79, 1167-1236.

[5] Ross, S., Gordon, G., & Bagnell, D. (2011, June). A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics (pp. 627-635). JMLR Workshop and Conference Proceedings.

[6] Shafiullah, N. M., Cui, Z., Altanzaya, A. A., & Pinto, L. (2022). Behavior transformers: Cloning $ k $ modes with one stone. Advances in neural information processing systems, 35, 22955-22968.

[7] Peng, X. B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018, May). Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA) (pp. 3803-3810). IEEE.

[8] Rajeswaran, A., Ghotra, S., Ravindran, B., & Levine, S. (2016). Epopt: Learning robust neural network policies using model ensembles. arXiv preprint arXiv:1610.01283.

Questions for authors

see weaknesses

1 comment

Thank you for the constructive review. We agree with the positioning point and will revise the paper accordingly.

Writing. We agree. We will make the writing clearer and easier to read, using standard terms.

Five seeds. We extended the matched Concat/FiLM study to five BC seeds and tested across seeds. The assignment effect is consistent. Switching Own to Best moves the Concat−FiLM gap toward Concat in every seed in HalfCheetah and Ant, significantly across seeds (+215, 95% CI [+96, +334]; +169, [+17, +322]). In both tasks the shift exceeds the original gap between the architectures (−160 and −71), and in HalfCheetah it flips the ranking significantly (Own −160 [−272, −48]; Best +55 [+5, +106]).

Positioning (contextual MDPs, CARL). We will formalize the setting as a contextual MDP and add this literature, including CARL, to related work. CARL shows that whether context helps depends on the training distribution and the evaluation region. Our message is that, when a context-conditioned policy is distilled from per-context teachers, it also depends on which teacher labels each context. This choice does not exist for RL-trained agents, and its effect is large: it shifts architecture comparisons by more than the gap between the architectures and can reverse them. We also explain where it matters, within the cMDP framework. Proposition 1 shows that if two neighboring contexts at distance w carry labels that differ by J at a state, a student that fits both within ε must respond to the context with slope at least (J − 2ε)/w, so label gaps force a steep context dependence between those settings. Empirically, this is where accurate context hurts (AUC 0.89), and changing only the labels removes or creates these losses. Practically, for anyone distilling specialists into one conditioned policy, this means that architecture claims must report the teacher assignment, and that label conflicts can be checked and fixed before training.

Domain randomization, multimodal BC and DAgger. Thank you; we will discuss all of these in related work. Domain randomization gains coverage by training one policy over sampled physics, multimodal BC models demonstrations with several action modes, and DAgger corrects the learner's state distribution. Our setting touches each of them: the student must generalize beyond its training physics, its labels conflict across neighboring settings, and it is tested on physics it was never trained on.

Mixture-of-Gaussians BC. We ran this ablation: a mixture-of-Gaussians head (K=5, trained by likelihood) on the same Concat and FiLM backbones, data and schedule (HalfCheetah and Walker2d, three seeds). Our findings hold regardless of the BC objective. Accurate physics still hurts between training settings: telling the student the true physics instead of the nominal physics lowers its return at the halfway settings by a median of 31% (Concat) and 33% (FiLM) in HalfCheetah, where all 12 MoG students show the loss, compared with 23% and 33% with MSE; in Walker2d, by 24% and 28%, compared with 27% and 27%. The teacher assignment also still matters: switching from Own to Best raises the return of every MoG student, by +378 to +757 in HalfCheetah and +160 to +517 in Walker2d.

Our test settings already vary parameters jointly: every one changes two or three parameters at once (the 13 settings of Section 3 and the 512 per task of Section 4). Under these joint variations, changing only the teacher reverses 49 of 160 architecture comparisons (Section 3); test sets drawn from the same distribution rank architectures in opposite orders in 12 of 24 comparisons (Section 4); and in HalfCheetah the true physics input lowers return on 64 multi-axis configurations by 20-31%. We will state this design in Section 2 and connect it to the results in Section 3.

Reviewer_Gt2LOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies phenomena that occur when trying to evaluate simulated locomotion policies (e.g., for a 2D walker). Rather than proposing a benchmark or running an evaluation of various models, the paper analyzes how fickle evaluations can be depending on a number of factors, with relative rankings flipping with seemingly innocuous changes. The concrete setting the authors study is in evaluating behavior cloned (from a PPO expert) policies for locomotion of e.g., a 2D walker/ant where the policies are conditioned on the physics parameters of the evaluation scene (e.g., gravity, joint friction, body mass) and the main thing being evaluated is the neural network architecture of the behavior cloned policy (e.g., where conditioning is done via concatenating to the state, or via FiLM). The innocuous changes in question that can affect (i.e., by flipping) rankings of different network architectures are (1) which expert is used for BC, either the PPO policy trained for that specific physics setting, or an expert trained for a different physics setting that happens to get higher return on this physics setting, and (2) the physics setting used as conditioning itself. For both (1) and (2) the authors run experiments finding that if you change the physics param conditioning used in evaluation, rankings of the different network architectures can change, and similarly if you change the expert policy, rankings can change. Then after demonstrating this, the authors run a number of other experiments that lead to the following findings. (1) Even independently sampled test sets from the same distribution can produce contradictory rankings. I.e., if the test set is too small, sampling variance can lead to different rankings on this test set. (2) the error bounds for evaluation is a function of the number of different physics params used during evaluation. I.e., if you want to convincingly show that one network architecture is better than another, you need to evaluate on a sufficiently large number of different possible physics settings, as the authors show that the error bound term includes in its expression the number of settings. (3) the authors develop a statistical predictor model for whether when small test sets are used, rankings might flip. This statistical model essentially predicts that architectures whose performance varies widely across different physics params are the ones likely to have different rankings for different small test sets (because variance is already high, and therefore under sampling error with small test sets you might get different rankings). (4) the authors find that sometimes better performance for an architecture can be obtained if the incorrect physics parameters are fed in as conditioning. The authors argue this happens because different experts (for different physics params) can have conflicting labels, and this can lead to a poorly trained BC policy that might erroneously do better when fed the wrong physics params at test time. The authors then do an elaborate casual intervention experiment to fix this issue. (5) Finally the authors propose an algorithm they call an Expert-by-Test Grid, which proposes that instead of using one expert or one physics parameter setting for evaluation, you should use multiple.

Critical strengths

(1) The authors correctly identify that using small test sets is not a good idea, because with sampling artifacts the conclusions drawn on this test set can be unreliable.

Critical weaknesses

(1) The paper is not easy to read for a very hard to explain reason. In my experience the reason the paper is hard to read is because of AI writing (I’ve had much past experience of similar hard to read writing that was hard to read because of AI). While the use of AI is not bad by itself, if it renders the paper hard to read, this is not a good thing.


(2) The premise of the paper is questionable, and there are a number of problems with the method and experiments. Listing them here:



Firstly, it is unclear to me why the authors are even studying the very specific setting of locomotion policies trained with behavior cloning, with evaluation conducted to identify better neural network architectures. This is very bespoke and not motivated in the paper.

Second, it is common practice when training locomotion policies for robots to train them with RL (e.g., a locomotion policy for a humanoid is trained with RL in sim and then transferred to a real humanoid). However the authors train their policies with behavior cloning off of a PPO trained policy, which does not make sense to me.

Third, why do the authors train separate PPO experts for different physics settings? Why not, as the students being distilled into, do the authors instead train a single expert with PPO that itself is conditionable on physics parameters? If the authors were to have done this, all of the problems the authors are trying to resolve are effectively obviated from the get go; there is no need to worry about which expert to distill because there’s no distillation happening at all; the RL trained policy is simply used directly. If a single physics param conditioned policy is trained with RL, there is no issue of different physics settings changing the ranking of network architectures; simply evaluate on a diverse enough set.

Fourth, why do the authors evaluate only on very simple locomotion domains, like hopper 2d and ant? It seems much more relevant to evaluate on real platforms used for locomotion today, like humanoids. This is a negative w.r.t. experimental thoroughness.

Fifth, in the experiments where the authors find that changing the expert policy induces a switch in rankings, there are flaws in the evaluation methodology that make this result questionable. The state distribution is kept the same, but only which policy labels the actions changes. But a different expert will induce a different state distribution, so this is not correct. Also, changing the expert to the one that doesn’t match the correct physics params but happens to perform better might lead to better performance for a particular architecture for another reason the authors didn’t consider: this different expert might be selected multiple times for different physics parameters, and so the student being distilled into has an easier problem of distilling fewer different policies.

Sixth, the finding that using a small test set can lead to different rankings because of sampling bias is not novel. This is a very well known phenomenon, and the solution is to use a larger test set if you can.

Seventh, a number of experiments done do not seem very useful. And their existence in this paper is not motivated. For example doing the variance analysis showing that the error bars in statistical predictions depend on how many different physics params are used in evaluation: this is obvious, and I don’t see what the problem is of just having many different physics parameters during evaluation. Another example is the whole causal intervention experiment the authors do to resolve conflicting expert labels when different experts are distilled into one physics parameter conditioned policy. If you simply train a single physics param conditioned PPO policy, none of this is an issue.

Finally eight, there is an issue with the high level premise. The premise/problem the authors, at a high level, are studying, is that under bad evaluation setups with insufficient varying of the parameters that matter, bad evaluation results can be obtained. But the solution is therefore tautological: simply fix the evaluation procedure to evaluate exactly what you care about. In fact the final method the authors come up with, the Expert-by-Test grid, is exactly this: evaluating on a sufficient number of physics settings. It is therefore unclear what the main novel thing the authors are presenting in their paper.

Questions for authors

No questions beyond what’s in the weaknesses section.

2 comments

Thank you for the detailed review.

(1) Readability. We agree. The main cause was inconsistent terminology. We are revising the paper to use one plain term per concept, defined where it first appears.

Our work is a study for judging in advance whether a comparison between controller designs can be trusted. When one robot controller is built by copying several expert controllers, two rarely reported choices, which expert is copied for each condition and which conditions are used for testing, can change which design looks best.

(2.1) Why study this setting? Because many robot controllers are built this way, and the comparisons used to pick their design can be wrong. Distilling per-condition RL specialists into one controller is standard in locomotion (Robot Parkour Learning, CoRL 2023; Barkour, 2023; Parkour in the Wild, Rudin et al., 2025; X-Loco for humanoids, Wang et al., 2026). These pipelines fix which specialist labels each condition, but the specialist trained at a condition is often not the best teacher there (33 of 44 settings in our banks), and changing only this choice reverses 49 of 160 design comparisons. We will add this motivation to the introduction.

(2.2–2.3) Why distill instead of training one physics-conditioned PPO policy? We do use RL: every teacher is trained with PPO. Training one specialist per condition and distilling them is the standard choice when conditions differ strongly. One RL policy over all conditions is then harder to optimize: the conditions compete for the same parameters, exploration must cover all of them at once, and the policy can settle on a compromise that is weak in each. Specialists avoid this, as argued by Divide-and-Conquer RL (Ghosh et al., 2018) and Rudin et al. (2025). Distillation by BC then merges them into one controller. BC is also offline: the student learns from a fixed dataset without new interaction, as do most large robot policies today (e.g., Octo, "Octo: An Open-Source Generalist Robot Policy", 2024). This also suits analysis: we can hold the states fixed and change only the labels or only the architecture, which RL training does not allow.

(2.4) Simple domains. We chose standard MuJoCo tasks because the study needs full cross-evaluation (11 teachers × 11 settings per task) and 98,304 episodes for the test-set study alone. The pipeline itself is used on humanoids (X-Loco, Wang et al., 2026), and our checks need only the trained teachers and students, so they apply there unchanged. We leave a full evaluation on humanoid platforms to future work.

(2.5) Fixed states and fewer teachers. Fixing the states is deliberate: if each assignment collected its own states, the label source and the state distribution would change together, and the cause of a reversal could not be identified. We also tested whether fewer teachers alone explains the effect. We kept Best's sharing structure (3 teachers for 11 settings) but used teachers that are not the best. The ranking does not reverse in any of three seeds (Concat − FiLM = +2, −61 and −91, against +83, +89 and +58 under Best). Fewer teachers alone does not explain the reversal; the quality of the shared teachers does. We will add this control to Section 3.3.

(2.6) Small test sets. The problem Section 4.2 identifies is not the size of the test set but how its uncertainty is reported. The usual interval treats the tested physics as fixed and covers only episode noise. By Eq. 5, both parts of the variance shrink at the same rate as settings are added, so this interval overstates certainty by the same factor at any test-set size. On our 192 comparisons, the usual interval supports 100 rankings, while one that includes the physics variance supports 9. We therefore report the physics-aware interval, together with a risk score that flags fragile comparisons before testing, from separate data (AUC 0.81).

(2.7) Variance analysis and intervention. We do not say that many physics settings are a problem; Eq. 5 gives the interval that is valid at any test-set size. The intervention tests the mechanism causally: with states and training fixed, changing only the labels removes the predicted failures (12/12 intervals above zero) or creates one (3/3 below zero). For distilled controllers, a failure can thus be fixed by relabeling alone.

(2.8) Our contribution is pre-evaluation: telling whether a comparison can be trusted before you run it. Current pipelines assume that the specialist trained at a condition is the right teacher for it. It often isn't. Another specialist teaches that condition better, and this hidden choice can decide which design wins. Both this choice and the choice of test physics can be checked ahead of time. Before training, the action gap between neighboring teachers tells you where accurate physics inputs will hurt (AUC 0.89). Before testing, a risk score from separate data tells you which comparisons will reverse (AUC 0.81).

I thank the authors for their response. After reading through the response thoroughly, I have decided to keep my original score, because my original assessment of the paper still holds and the rebuttal response did not change the validity of any of my originally pointed out weaknesses. Please see my original review for my full opinion on the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.