- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors propose an idea of recovering teacher policies by using synthetic distributions. The idea is rather simple -- take pretrained teacher, sample a lot of synthetic data from a Gaussian (or any distribution) and just fit a student to the responses of the teacher.
The paper includes empirical results on 4 gym environments -- Hopper-v5, HalfCheetah-v5, Ant-v5, Humanoid-v5.
Critical strengths
The paper presents a simple idea and the empirical results look interesting, though more careful study is required.
Critical weaknesses
The paper is not well written and generally hard to follow. While I appreciate that the authors introduced the Section 3, which describes the setup, it is very hard to follow what happens after that. (Though I get the gist, but it is not written in a clear way). Moreover, the paper contains many sentences which look like AI slop, "...something is something rather than something..." -- very distinct style of LLM-based writing, despite the fact that the authors claimed not to have used AI for writing and only for formatting.
Here are a few examples (out of many many more) of where it is hard to understand what is going on:
- Authors say "broader r_{90} scale?". What does it mean?
- In section 5.2, the authors for some reason use top-78, why 78 and not 77? Similarly, in 5.3, they write eq.6 with V_{78}, where does it come from? There are no explanations about it in the main text.
I also have big concerns regarding all the experimental setup and the results. Intuitively, I can understand that the idea might work if the synthetic distribution covers well the mass under teacher trajectories -- in this case, one might hope to replicate the teacher. Here, using Gaussian seems to work extremely well, which is very strange. One explanation I can imagine is that the teachers which the authors are using are actually quite weak (in fact, 1706 on Hopper-v5, 3181 on HalfCheetah-v5, 4945 on Ant-v5 and 6488 on Humanoid-v5 is rather weak). Imagine of the teacher policies are not doing much -- they move in the space, but not really solving the task. In this case most of their trajectories might not be that far away from a Gaussian distribution. On many tasks, the starting point is like 40% performance of the teacher, which suggests that 40% of teacher performance might be very close to random policy.
In order to strengthen the results of the paper, I suggest the following study:
-
For each task, you can record "weak", "medium" and "strong" teacher. You need to make sure that "strong" actually solves the task -- if you visually inspect the policies, they should be valid.
-
Run your method in each scenario. My prediction is that as teacher becomes stronger -- its distribution will move much further away from random policy and from a Gaussian, and the effectiveness of this method will significantly deterioriate.
Another critical weakness -- the method is only test on 4 gym environments. These environments use the same backend and have similar observation normalization strategies. I think results in a very different regime will significantly strengthen the paper.
As it stands now, I recommend rejection, but I am open for discussion during rebuttal.
Questions for authors
See above
6 comments
The table below reports Nearest distance is the median, for each teacher, from 10,000 normalized rollout states to the nearest of 100,000 Gaussian samples.
| Environment | Nearest Gaussian distance | Rollout vs. Gaussian energy | Gaussian vs. Gaussian energy | Relative energy |
|---|---|---|---|---|
| Hopper | 1.175 ± 0.040 | 0.304 ± 0.061 | 0.0009 ± 0.0001 | 358 ± 93× |
| HalfCheetah | 2.412 ± 0.125 | 0.359 ± 0.128 | 0.0012 ± 0.0003 | 326 ± 139× |
| Walker2d | 2.201 ± 0.167 | 0.354 ± 0.122 | 0.0011 ± 0.0002 | 346 ± 136× |
| Humanoid | 20.469 ± 0.336 | 1.822 ± 0.492 | 0.0054 ± 0.0001 | 336 ± 94× |
| Ant | 7.808 ± 0.068 | 1.792 ± 0.250 | 0.0030 ± 0.0001 | 598 ± 97× |
Relative energy is the rollout–Gaussian energy divided by the Gaussian–Gaussian reference, calculated separately for each teacher before averaging. This shows how far the true post-normalized rollout stats compare to true isotropic gaussian. Despite normalization, the rollout stats follow a different distribution and have little to no overlap with the gaussian distribution we sample synthetic queries from.
Additional environments, task & algorithms
We provide evidence from additional environments in Appendix E.1, Table 34, Appendix E.3, Table 36, and Appendix C Table 5. Additionally, we have added experiments on two more environments presented below. These experiments show diversity in terms of tasks, environment representations, and algorithms.
Appendix E.1 tests recovery for a ReLU LunarLander teacher, while Appendix C tests whether a safety-aware algorithm like PPO-Lag on SafetyPointGoal1 can also be recovered with synthetic queries. In both cases, we get strong recovery, showing that this framework goes beyond the Mujoco simulations presented in the paper. The SafetyPointGoal1 students also show similar safety performance, which indicates that the teacher's properties have been transferred to the student via synthetic queries. In the LunarLander experiment, orthogonal thickening raises Top-2 recovery from 52.0% at =0 to 89.9% at =0.10, with essentially unchanged sensitivity overlap. Sensitivity is computed from the continuous action logits before discrete action selection. We will make this experiment and its setup easier to locate in the main text.
We conduct additional recovery experiments on HighwayEnv parking and electric-motor current control. The parking teachers use SAC with hindsight experience replay (HER); the motor-control teachers use DDPG. Students are trained by supervised imitation of the teachers' pre-Tanh actor outputs, using the SAC mean as the parking target.
| Environment | Metric | Teacher | 100k student | 500k student |
|---|---|---|---|---|
| HighwayEnv parking | Success | 99.6 ± 0.9% | 59.2 ± 24.3% | 83.2 ± 7.7% |
| Collisions | 0.2 ± 0.4% | 39.0 ± 22.7% | 16.6 ± 7.7% | |
| Raw return | -6.88 ± 0.30 | -10.45 ± 2.60 | -8.08 ± 0.51 | |
| Motor current control | Tracking MAE | 0.0786 ± 0.0093 | 0.0834 ± 0.0106 | 0.0793 ± 0.0092 |
| Limit violations | 1.4 ± 1.3% | 1.2 ± 0.8% | 1.8 ± 0.8% | |
| Raw return | -7.64 ± 0.91 | -8.09 ± 1.07 | -7.70 ± 0.90 |
The parking teacher succeeds in 99.6% of episodes, providing a task-specific measure of teacher competence. Increasing the student query budget raises success from 59.2% to 83.2% and lowers collisions from 39.0% to 16.6%. For motor control, the 500k student's mean tracking MAE is 0.0793 against the teacher's 0.0786, a difference of approximately 0.9%; its limit-violation rate is 1.8% against 1.4%.
The parking actors have three 256-unit ReLU hidden layers. The motor actors have two 100-unit hidden layers with ReLU and LeakyReLU(0.2). These experiments therefore extend the evaluation to goal-conditioned driving and electrical current regulation, using off-policy teachers and ReLU-family actors.
We also run the recovery setup with the MNIST dataset. In Appendix E.3, we show that a network trained on the MNIST dataset can be completely recovered with synthetic queries, and the students show similar degradation when labels are shuffled. Synthetic queries in the Image domain correspond to random noise that are not similar to the actual digits. Despite that, the student recovers a substantial fraction of the teacher and matches the test metrics. This experiment also shows that if the teacher memorizes the solutions, as in the shuffled label regime, synthetic queries fail to recover the teacher.
All experiments above show diversity in terms of tasks, observation types and algorithms. We are happy to perform additional experiments if requested by the reviewer.
Clarification of broader r_{90} and top 78
For eigenvalues of the teacher sensitivity matrix , we define
r_{90} = \min\left\\{ k : \sum_{j=1}^{k}\lambda_j \geq 0.90\sum_{j=1}^{d}\lambda_j \right\\}
The phrase “broader scale” refers to the larger subspace accounting for approximately 90% of average sensitivity. In the original Humanoid cohort, mean r90 is approximately 78, which motivated the fixed Top-78 intervention. V78 contains the corresponding 78 leading eigenvectors. The rank is fixed across the compared conditions; it is not exactly every teacher's individual . There is no separate theoretical significance to choosing 78 instead of 77.
We will replace the ambiguous phrase, introduce and before the interventions, and state how the fixed ranks were selected.
Clarity
We will work to improve clarity by addressing the ambiguity raised in this review. We stand by our AI use statement. While it is impossible to conclusively prove that this paper is not AI generated, we would like to highlight the novelty of our work as evidence of human creation.
To the best of our knowledge we are the first to show that purely synthetic, noise based, data is sufficient to recover a teacher’s policy. Additionally, this paper explores the theoretical justification for this approach instead of merely presenting this phenomenon empirically. While this approach is intuitive, it is still novel and we are providing substantial theoretical and empirical evidence to justify its deployment in imitation learning pipelines broadly.
We thank the reviewer for their feedback and suggestions. We address them below.
Teacher Performance:
The scores achieved by the teachers presented in the paper correspond to success at the task, which can be seen in the videos at this anonymized URL: https://anonymous.4open.science/r/geometric-coverage-7B20/site/index.html
These environments don’t have a clear “solve” criteria because theoretically the agent can always go faster, which would lead to a higher score. However, we followed your suggestion about creating stronger teachers to test whether the same recovery conditions hold. So, we extended the training beyond the initial task completion budget. The new budget and network settings are reported below, after extensive hyperparameter tuning. These extended budgets allowed us to train stronger teachers which show even stronger evidence in support of our hypothesis (for example: Humanoid at 88.6% recovery with 500K queries). This improvement is aligned with the hypothesis, since extended training of the teacher is creating smoother policies, showing stronger generalization over memorization, and are hence easier to recover with synthetic data.
We trained 5 teachers (5 seeds) per environment, and 2 students, one at 100K sampled gaussian queries and one at 500K queries. This makes a total of 10 students per environment and 50 students across this experiment. All students use full width networks.
We see substantial recovery of the teacher at 500K queries. The results for the new stronger teachers and their corresponding students outperform the results reported in our paper, and the recovery remains the same or is even better for Humanoid and Ant at 500K queries. Qualitative analysis of the trajectories shows that both teachers and students solve the task effectively and maintain similar performance. We provide videos as evidence with the above anonymized URL.
We also repeated the subspace intervention with five stronger Humanoid teachers. Orthogonal thickening increased recovery from 61.2 ± 12.9% to 97.0 ± 3.0%. With covariance eigenvalues, budgets, and training settings matched, thickened random and bottom-subspace queries achieved 59.0 ± 21.1% and 20.6 ± 17.2%, respectively. The geometry effect therefore persists in this stronger-teacher cohort.
The newer stronger teachers have a lower number of sensitive directions, which correspond to higher recovery as demonstrated earlier.
Thus, we have shown that both weak to medium teachers and stronger teachers can be recovered with our proposed framework. We will provide a link to these demonstrations in the revision. We thank the reviewer for suggesting this additional experiment, we believe it substantially strengthens practical recovery evidence.
Gaussian Overlap Possibility
We address the possibility of an overlap with the gaussian in the main paper (Section 3, line 170-174) and with more details in Appendix D.9.
We have shown in the paper (section 3 & appendix D.9) that the normalized state-space after training does not overlap with standard gaussian distributions. The VecNormalize wrapper lags behind and thus the true trajectory is not centered at 0. Our gaussian queries are sampled from the standard normal and show substantial recovery despite being further away from the true trajectory distribution. Weak/Strong teachers do not affect the performance of the proposed synthetic-sampling framework. These experiments characterize how query geometry affects closed-loop recovery. They do not require synthetic observations to be physically valid or the rollout distribution to be Gaussian. Our contribution studies the transfer conditions using a simple distillation procedure. To emphasize this further, we will move parts of Appendix D.9 to the main paper in the final version of the paper (which is allowed an additional page).