acceptodds
Under review as a conference paper at ICLR 2027

On the Structure of Reinforcement Learning Policies: Recovery from Synthetic Observations via Geometric Coverage

Abstract

Can a trained reinforcement learning policy be recovered without trajectories, rewards, or valid environment states? Across control tasks, students trained only on teacher-labeled synthetic inputs recover substantial closed-loop performance. Recovery is not explained by physical realism alone. Instead, it depends on two geometric properties of the query distribution: excitation of teacher-sensitive directions and sufficient variance outside the dominant sensitive subspace. Matched subspace interventions isolate directional excitation, while orthogonal thickening isolates ambient support; on Humanoid-v5, -scale orthogonal noise raises recovery from to with almost unchanged sensitivity overlap. We show theoretically that covariance mismatch governs transfer from query-space error to deployment error and that strict subspace queries can leave deployment behavior unidentifiable. A synthetic-only pilot further shows that inverse-covariance coverage can diagnose failures missed by raw overlap, although it is not universally predictive. These effects replicate for SAC and discrete-action ReLU policies. Small real-state buffers provide complementary deployment information, and action-imitation accuracy can continue improving after closed-loop return saturates. Together, these results identify geometric coverage as a central factor governing passive recovery of state-based policies.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Knowledge Distillation, Offline Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5aBmOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2(+1)
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose an idea of recovering teacher policies by using synthetic distributions. The idea is rather simple -- take pretrained teacher, sample a lot of synthetic data from a Gaussian (or any distribution) and just fit a student to the responses of the teacher.

The paper includes empirical results on 4 gym environments -- Hopper-v5, HalfCheetah-v5, Ant-v5, Humanoid-v5.

Critical strengths

The paper presents a simple idea and the empirical results look interesting, though more careful study is required.

Critical weaknesses

The paper is not well written and generally hard to follow. While I appreciate that the authors introduced the Section 3, which describes the setup, it is very hard to follow what happens after that. (Though I get the gist, but it is not written in a clear way). Moreover, the paper contains many sentences which look like AI slop, "...something is something rather than something..." -- very distinct style of LLM-based writing, despite the fact that the authors claimed not to have used AI for writing and only for formatting.

Here are a few examples (out of many many more) of where it is hard to understand what is going on:

  1. Authors say "broader r_{90} scale?". What does it mean?
  2. In section 5.2, the authors for some reason use top-78, why 78 and not 77? Similarly, in 5.3, they write eq.6 with V_{78}, where does it come from? There are no explanations about it in the main text.

I also have big concerns regarding all the experimental setup and the results. Intuitively, I can understand that the idea might work if the synthetic distribution covers well the mass under teacher trajectories -- in this case, one might hope to replicate the teacher. Here, using Gaussian seems to work extremely well, which is very strange. One explanation I can imagine is that the teachers which the authors are using are actually quite weak (in fact, 1706 on Hopper-v5, 3181 on HalfCheetah-v5, 4945 on Ant-v5 and 6488 on Humanoid-v5 is rather weak). Imagine of the teacher policies are not doing much -- they move in the space, but not really solving the task. In this case most of their trajectories might not be that far away from a Gaussian distribution. On many tasks, the starting point is like 40% performance of the teacher, which suggests that 40% of teacher performance might be very close to random policy.

In order to strengthen the results of the paper, I suggest the following study:

  1. For each task, you can record "weak", "medium" and "strong" teacher. You need to make sure that "strong" actually solves the task -- if you visually inspect the policies, they should be valid.

  2. Run your method in each scenario. My prediction is that as teacher becomes stronger -- its distribution will move much further away from random policy and from a Gaussian, and the effectiveness of this method will significantly deterioriate.

Another critical weakness -- the method is only test on 4 gym environments. These environments use the same backend and have similar observation normalization strategies. I think results in a very different regime will significantly strengthen the paper.

As it stands now, I recommend rejection, but I am open for discussion during rebuttal.

Questions for authors

See above

6 comments

We thank the reviewer for their feedback and suggestions. We address them below.

Teacher Performance:

The scores achieved by the teachers presented in the paper correspond to success at the task, which can be seen in the videos at this anonymized URL: https://anonymous.4open.science/r/geometric-coverage-7B20/site/index.html

These environments don’t have a clear “solve” criteria because theoretically the agent can always go faster, which would lead to a higher score. However, we followed your suggestion about creating stronger teachers to test whether the same recovery conditions hold. So, we extended the training beyond the initial task completion budget. The new budget and network settings are reported below, after extensive hyperparameter tuning. These extended budgets allowed us to train stronger teachers which show even stronger evidence in support of our hypothesis (for example: Humanoid at 88.6% recovery with 500K queries). This improvement is aligned with the hypothesis, since extended training of the teacher is creating smoother policies, showing stronger generalization over memorization, and are hence easier to recover with synthetic data.

SettingHopperWalker2dHalfCheetahAntHumanoid
Training budget per seed5M steps10M10M20M30M
Actor hidden layers256, 256256, 256256, 256256, 256512, 512
Critic hidden layers256, 256256, 256256, 256256, 256512, 512
ActivationTanhTanhReLUTanhTanh

We trained 5 teachers (5 seeds) per environment, and 2 students, one at 100K sampled gaussian queries and one at 500K queries. This makes a total of 10 students per environment and 50 students across this experiment. All students use full width networks.

EnvironmentNew teacherStudent 100k queriesStudent 500k queriesRecovery at 500k
Hopper3,602 ± 793,538 ± 933,579 ± 10699.4%
Walker2d5,529 ± 7084,221 ± 1,9405,444 ± 63698.5%
HalfCheetah5,741 ± 6184,326 ± 1,0795,235 ± 49991.2%
Ant5,523 ± 2635,109 ± 4535,401 ± 24597.8%
Humanoid8,697 ± 3594,115 ± 1,6767,705 ± 89788.6%

We see substantial recovery of the teacher at 500K queries. The results for the new stronger teachers and their corresponding students outperform the results reported in our paper, and the recovery remains the same or is even better for Humanoid and Ant at 500K queries. Qualitative analysis of the trajectories shows that both teachers and students solve the task effectively and maintain similar performance. We provide videos as evidence with the above anonymized URL.

We also repeated the subspace intervention with five stronger Humanoid teachers. Orthogonal thickening increased recovery from 61.2 ± 12.9% to 97.0 ± 3.0%. With covariance eigenvalues, budgets, and training settings matched, thickened random and bottom-subspace queries achieved 59.0 ± 21.1% and 20.6 ± 17.2%, respectively. The geometry effect therefore persists in this stronger-teacher cohort.

EnvironmentOlder d_{sens}Newer d_{sens}
Hopper3.83 ± 0.313.53 ± 0.72
HalfCheetah4.66 ± 0.684.52 ± 1.66
Walker2d4.90 ± 0.463.98 ± 1.06
Ant7.97 ± 0.775.36 ± 0.90
Humanoid14.96 ± 1.196.59 ± 0.64

The newer stronger teachers have a lower number of sensitive directions, which correspond to higher recovery as demonstrated earlier.

Thus, we have shown that both weak to medium teachers and stronger teachers can be recovered with our proposed framework. We will provide a link to these demonstrations in the revision. We thank the reviewer for suggesting this additional experiment, we believe it substantially strengthens practical recovery evidence.

Gaussian Overlap Possibility

We address the possibility of an overlap with the gaussian in the main paper (Section 3, line 170-174) and with more details in Appendix D.9.

We have shown in the paper (section 3 & appendix D.9) that the normalized state-space after training does not overlap with standard gaussian distributions. The VecNormalize wrapper lags behind and thus the true trajectory is not centered at 0. Our gaussian queries are sampled from the standard normal and show substantial recovery despite being further away from the true trajectory distribution. Weak/Strong teachers do not affect the performance of the proposed synthetic-sampling framework. These experiments characterize how query geometry affects closed-loop recovery. They do not require synthetic observations to be physically valid or the rollout distribution to be Gaussian. Our contribution studies the transfer conditions using a simple distillation procedure. To emphasize this further, we will move parts of Appendix D.9 to the main paper in the final version of the paper (which is allowed an additional page).

The table below reports Nearest distance is the median, for each teacher, from 10,000 normalized rollout states to the nearest of 100,000 Gaussian samples.

EnvironmentNearest Gaussian distanceRollout vs. Gaussian energyGaussian vs. Gaussian energyRelative energy
Hopper1.175 ± 0.0400.304 ± 0.0610.0009 ± 0.0001358 ± 93×
HalfCheetah2.412 ± 0.1250.359 ± 0.1280.0012 ± 0.0003326 ± 139×
Walker2d2.201 ± 0.1670.354 ± 0.1220.0011 ± 0.0002346 ± 136×
Humanoid20.469 ± 0.3361.822 ± 0.4920.0054 ± 0.0001336 ± 94×
Ant7.808 ± 0.0681.792 ± 0.2500.0030 ± 0.0001598 ± 97×

Relative energy is the rollout–Gaussian energy divided by the Gaussian–Gaussian reference, calculated separately for each teacher before averaging. This shows how far the true post-normalized rollout stats compare to true isotropic gaussian. Despite normalization, the rollout stats follow a different distribution and have little to no overlap with the gaussian distribution we sample synthetic queries from.

Additional environments, task & algorithms

We provide evidence from additional environments in Appendix E.1, Table 34, Appendix E.3, Table 36, and Appendix C Table 5. Additionally, we have added experiments on two more environments presented below. These experiments show diversity in terms of tasks, environment representations, and algorithms.

Appendix E.1 tests recovery for a ReLU LunarLander teacher, while Appendix C tests whether a safety-aware algorithm like PPO-Lag on SafetyPointGoal1 can also be recovered with synthetic queries. In both cases, we get strong recovery, showing that this framework goes beyond the Mujoco simulations presented in the paper. The SafetyPointGoal1 students also show similar safety performance, which indicates that the teacher's properties have been transferred to the student via synthetic queries. In the LunarLander experiment, orthogonal thickening raises Top-2 recovery from 52.0% at =0 to 89.9% at =0.10, with essentially unchanged sensitivity overlap. Sensitivity is computed from the continuous action logits before discrete action selection. We will make this experiment and its setup easier to locate in the main text.

We conduct additional recovery experiments on HighwayEnv parking and electric-motor current control. The parking teachers use SAC with hindsight experience replay (HER); the motor-control teachers use DDPG. Students are trained by supervised imitation of the teachers' pre-Tanh actor outputs, using the SAC mean as the parking target.

EnvironmentMetricTeacher100k student500k student
HighwayEnv parkingSuccess99.6 ± 0.9%59.2 ± 24.3%83.2 ± 7.7%
Collisions0.2 ± 0.4%39.0 ± 22.7%16.6 ± 7.7%
Raw return-6.88 ± 0.30-10.45 ± 2.60-8.08 ± 0.51
Motor current controlTracking MAE0.0786 ± 0.00930.0834 ± 0.01060.0793 ± 0.0092
Limit violations1.4 ± 1.3%1.2 ± 0.8%1.8 ± 0.8%
Raw return-7.64 ± 0.91-8.09 ± 1.07-7.70 ± 0.90

The parking teacher succeeds in 99.6% of episodes, providing a task-specific measure of teacher competence. Increasing the student query budget raises success from 59.2% to 83.2% and lowers collisions from 39.0% to 16.6%. For motor control, the 500k student's mean tracking MAE is 0.0793 against the teacher's 0.0786, a difference of approximately 0.9%; its limit-violation rate is 1.8% against 1.4%.

The parking actors have three 256-unit ReLU hidden layers. The motor actors have two 100-unit hidden layers with ReLU and LeakyReLU(0.2). These experiments therefore extend the evaluation to goal-conditioned driving and electrical current regulation, using off-policy teachers and ReLU-family actors.

We also run the recovery setup with the MNIST dataset. In Appendix E.3, we show that a network trained on the MNIST dataset can be completely recovered with synthetic queries, and the students show similar degradation when labels are shuffled. Synthetic queries in the Image domain correspond to random noise that are not similar to the actual digits. Despite that, the student recovers a substantial fraction of the teacher and matches the test metrics. This experiment also shows that if the teacher memorizes the solutions, as in the shuffled label regime, synthetic queries fail to recover the teacher.

All experiments above show diversity in terms of tasks, observation types and algorithms. We are happy to perform additional experiments if requested by the reviewer.

Clarification of broader r_{90} and top 78

For eigenvalues λ1≥…≥λd\lambda_1 \geq … \geq \lambda_d of the teacher sensitivity matrix G=E[J(x)⊤J(x)]G=\mathbb E[J(x)^\top J(x)], we define

r_{90} = \min\left\\{ k : \sum_{j=1}^{k}\lambda_j \geq 0.90\sum_{j=1}^{d}\lambda_j \right\\}

The phrase “broader r90r_{90} scale” refers to the larger subspace accounting for approximately 90% of average sensitivity. In the original Humanoid cohort, mean r90 is approximately 78, which motivated the fixed Top-78 intervention. V78 contains the corresponding 78 leading eigenvectors. The rank is fixed across the compared conditions; it is not exactly every teacher's individual r90r_{90}. There is no separate theoretical significance to choosing 78 instead of 77.

We will replace the ambiguous phrase, introduce r90r_{90} and VkV_k before the interventions, and state how the fixed ranks were selected.

Clarity

We will work to improve clarity by addressing the ambiguity raised in this review. We stand by our AI use statement. While it is impossible to conclusively prove that this paper is not AI generated, we would like to highlight the novelty of our work as evidence of human creation.

To the best of our knowledge we are the first to show that purely synthetic, noise based, data is sufficient to recover a teacher’s policy. Additionally, this paper explores the theoretical justification for this approach instead of merely presenting this phenomenon empirically. While this approach is intuitive, it is still novel and we are providing substantial theoretical and empirical evidence to justify its deployment in imitation learning pipelines broadly.

Reviewer_smNpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether a trained RL policy can be recovered from fixed synthetic observations labeled by a teacher, without trajectories, rewards, or environment interaction during student training. The authors show substantial recovery on several control tasks and argue that recovery depends on exciting teacher-sensitive directions and keeping enough query variance outside the dominant sensitivity subspace. They test this with matched subspace interventions, orthogonal thickening, and Gaussian queries informed by rollout statistics, and they add linear-model results on subspace identifiability and error transfer. Additional experiments cover pilot-based query design, real/synthetic mixing, and extensions beyond PPO.

Critical strengths
  1. The recovery setting is interesting. Learning useful policies from synthetic observations without trajectories or reward feedback offers a valuable perspective on the structure of trained policies. Potential suggestion see Questions No.5.

  2. The controlled subspace and thickening experiments provide credible evidence that both directional sensitivity and broader coverage affect recovery. The replication with SAC and discrete action ReLU policies strengthens this finding.

Critical weaknesses
  1. The benefit of sensitivity alignment under full support remains unresolved. The paper does not compare thickened sensitive, random, and insensitive subspaces with matched covariance eigenvalues. The interaction between directional alignment and orthogonal coverage therefore remains incompletely characterized.

  2. The attribution of covariance gains to correlations is not adequately supported. Replacing identity covariance with empirical covariance changes both marginal variances and correlations. A diagonal covariance control is needed to separate these contributions.

  3. The applicability of the inverse covariance diagnostic remains unclear. It predicts action errors in several environments but fails on Humanoid, which provides the strongest thickening result. The paper does not identify measurable conditions that explain this difference

Questions for authors
  1. Could the authors compare thickened sensitive, random, and insensitive subspaces with the same positive, nonuniform covariance eigenvalues, query budgets, and training settings?

  2. Could the covariance experiment include a diagonal empirical covariance baseline at the same mean? This would clarify how much improvement comes specifically from correlations.

  3. What observable properties of the policy or rollout distribution explain when the inverse covariance diagnostic succeeds or fails?

  4. Could the authors examine imitation errors on states visited by the student? This would help connect the existing diagnostics on teacher rollout states to the errors encountered during student execution.

  5. Additional question/suggestion motivated by personal interest. Can recovery be understood through the row space and null space of the teacher Jacobian, together with the range and null space of the query covariance? How do the relationships between these subspaces explain successful recovery and the benefit of orthogonal thickening? Since the sensitivity matrix averages Jacobian Gram matrices across rollout states, how should this interpretation account for variation in the Jacobian subspaces across states?

I would consider raising my score if the authors address these concerns through convincing clarification and supporting evidence.

5 comments

We thank the reviewer for their review and suggestions. We have run additional experiments and provide clarifications below.

We added the matched full-support and diagonal-covariance controls suggested in the review. Following reviewer 5aBm's suggestion to examine stronger policies, we also trained five additional higher-return teachers per environment and new Gaussian-query students at 100k and 500k budgets. At 500k queries, mean recovery is 99.4% on Hopper, 98.5% on Walker2d, 91.2% on HalfCheetah, 97.8% on Ant, and 88.6% on Humanoid. We provide more details in their rebuttal.

The covariance and orientation controls below use the original ten-teacher cohorts. The inverse-coverage follow-up uses the new Hopper and Humanoid teachers, with 40 additional students trained specifically for that comparison.

W1 & Q1. Benefit of sensitivity alignment.

Sensitive, random, and insensitive query distributions have identical positive covariance eigenvalues, full support, query budgets, and training settings. For top-sensitive, random, and bottom-sensitive (in-sensitive) rank-kk projectors PP, we use Gaussian queries with

Σq=P+ε2(I−P),ε=0.10.\Sigma_q=P+\varepsilon^2(I-P),\qquad \varepsilon=0.10.

Each condition has kk covariance eigenvalues equal to 1 and d−kd-k equal to 0.01. Covariance spectra, total variance, and full support are therefore identical; orientation changes. We use k=17k=17 on Ant and k=78k=78 on Humanoid. All conditions receive 500k queries and 100 training epochs, with paired Gaussian draws, student initialization, and minibatch ordering within teacher.

EnvironmentTop minus random: gain in pp [95% CI]Top minus bottom: gain in pp [95% CI]
Ant+15.59 [5.53, 25.55]+68.10 [54.98, 79.40]
Humanoid+18.41 [12.02, 24.55]+36.80 [30.65, 41.66]

Top orientation also gives lower mean action MSE on teacher-visited and student-visited states, for both raw actor outputs and executed actions. This directly establishes an orientation benefit after orthogonal coverage is matched.

Adding orthogonal variance to strict-top queries also raises recovery from 23.07% to 86.18% on Ant and from 10.12% to 81.41% on Humanoid in this experiment. The controls therefore support both the benefit of orthogonal coverage and the additional role of orientation. We will add this to the paper so that the characterization is complete.

We repeated the subspace intervention with five stronger Humanoid teachers. Recovery increased from 61.2 ± 12.9% under strict sensitive-subspace queries to 97.0 ± 3.0% after orthogonal thickening. With matched positive covariance eigenvalues, budgets, and training settings, thickened random and bottom-subspace queries achieved 59.0 ± 21.1% and 20.6 ± 17.2%, respectively. These results support a benefit from sensitivity alignment even when all query distributions have full support.

2. Does full covariance help beyond marginal variance adaptation?

We compared N(m,I)\mathcal N(m,I), N(m,diag⁡Σ^)\mathcal N(m,\operatorname{diag}\hat\Sigma), and N(m,Σ^)\mathcal N(m,\hat\Sigma) at both m=0m=0 and m=μ^m=\hat\mu. Diagonal and full covariance use the same estimate, preserving every marginal variance and total variance. This separate study uses 100k training queries and 20 epochs per condition.

EnvironmentFull minus diagonal, zero mean: pp [95% CI]Full minus diagonal, empirical mean: pp [95% CI]
Ant+10.77 [6.22, 15.61]+6.48 [3.25, 10.18]
Humanoid+34.29 [28.63, 39.60]+32.82 [24.34, 41.52]

All four primary comparisons survive Holm correction (padj=0.007812p_{\mathrm{adj}}=0.007812). Mean action MSE also decreases on teacher and student states. Marginal scaling contributes, particularly on Ant; the full-minus-diagonal contrast establishes a further benefit from correlations in the tested policy coordinates.

Both control studies use the ten original teachers per environment and 100 evaluation episodes per policy.

3. What explains the environment-dependent performance of inverse coverage?

The stronger-teacher follow-up provides evidence that held-out query fit supplies information missing from coverage alone. For each of five new teachers per environment, we train four students using one fixed synthetic-pilot subspace and ε∈{0.10,0.25,0.50,1.00}\varepsilon\in\{0.10,0.25,0.50,1.00\}. Query-covariance trace is fixed across conditions. Each student receives 99k training labels after a charged 1k-location pilot.

Let LqL_q be raw actor MSE on 4,096 fresh query-distribution inputs, used only for diagnostics. We calculate correlations with independently measured teacher-state raw MSE within each teacher, then summarize across the five teachers:

EnvironmentMean ρ\rho: RinvR_{\mathrm{inv}}Mean ρ\rho: LqL_qMean ρ\rho: RinvLqR_{\mathrm{inv}}L_q
Hopper1.00−0.401.00
Humanoid−0.120.480.88

Coverage alone correctly orders the four Hopper errors for every teacher and has no positive association for any Humanoid teacher. On Humanoid, mean query error increases from 0.0264 to 0.1267 between ε=0.10\varepsilon=0.10 and 1.00, while inverse coverage decreases from 700.8 to 249.8. The coverage score omits this substantial change in how accurately the student fits its query distribution.

The predeclared exploratory product proxy has positive Humanoid correlations for all five teachers, ranging from 0.8 to 1.0. Exploratory prediction with complete teachers held out gives log-MSE R2=0.443R^2=0.443 for the product, compared with 0.122 for query error alone and −0.118 for coverage alone. Hopper gains no predictive benefit from adding query error. These five-fold estimates do not establish statistically significant model superiority.

We also corrected the historical centered-covariance implementation to use deployment second moments; the earlier Humanoid failure persists after that correction. The new findings support considering query fit alongside coverage, consistent with the transfer analysis’s dependence on both quantities. This remains a post-training diagnostic and an empirical lead on the failure, rather than a general applicability criterion. The study does not isolate approximation from optimization error or establish a causal effect of teacher strength.

4. What happens on states visited by the student?

We evaluated the 40 newly trained students using five higher-return teachers each for Hopper and Humanoid, with four query-covariance conditions per teacher. Each policy received 100 evaluation episodes. Both MSE columns measure disagreement between student and teacher; they differ in which policy visited the input states.

EnvironmentεMSE on teacher statesMSE on student statesPaired difference [95% CI]Recovery (%)
Hopper0.100.089200.08748−0.00172 [−0.01176, 0.00846]56.86
Hopper0.250.020750.02058−0.00017 [−0.00242, 0.00110]72.80
Hopper0.500.006780.00714+0.00036 [0.00001, 0.00073]90.11
Hopper1.000.003120.00323+0.00010 [−0.00001, 0.00021]96.20
Humanoid0.100.137500.14901+0.01150 [−0.00178, 0.02324]57.26
Humanoid0.250.121500.12682+0.00532 [−0.00199, 0.01361]64.14
Humanoid0.500.131770.13607+0.00430 [−0.00072, 0.00932]54.04
Humanoid1.000.211470.20357−0.00790 [−0.02384, 0.00746]35.56

MSE entries are means across five teachers. Differences are student-state minus teacher-state MSE, with exploratory, pointwise 95% paired teacher-bootstrap intervals from 20,000 resamples. Recovery is a ratio of cohort mean returns.

Mean disagreement is higher on student states in some conditions and lower in others. These measurements show condition-dependent differences without supporting a universal increase in error on student-visited states.

5. How do Jacobian row/null spaces interact with query support?

Thank you for suggesting this interpretation. It clarifies the distinct roles of sensitivity alignment and orthogonal coverage. At a given state, the Jacobian row space contains locally sensitive input directions, while its null space contains directions with no first-order effect on the teacher’s output. Query covariance determines which directions receive supervision. Aligning query variance with sensitive directions exposes action variation, but strict-subspace queries leave student behavior outside their support unconstrained.

This remains important even when an omitted direction is genuinely irrelevant to the teacher. For example, if the teacher is T(x1,x2)=x1T(x_1,x_2)=x_1 and queries satisfy x2=0x_2=0, students of the form x1+cx2x_1+cx_2 all fit perfectly. Orthogonal thickening supplies supervision that penalizes this unsupported dependence. State-dependent Jacobian subspaces are accounted for through v⊤Gv=Ex∥J(x)v∥2.v^\top Gv=\mathbb E_x\|J(x)v\|^2.

Thus, null⁡(G)\operatorname{null}(G) consists of directions annihilated by the Jacobian almost everywhere under the reference distribution. Its range aggregates sensitivity from potentially different local row spaces. Consequently, the leading eigenspace is an average sensitivity summary, and directions outside it can remain sensitive at particular states.

This interpretation explains why alignment and orthogonal coverage can both matter: informative directions reveal teacher variation, while broader supervision constrains student behavior beyond the selected subspace. The covariance-transfer result formalizes the coverage requirement for linear error; the matched interventions provide empirical evidence for the nonlinear policies studied here. We will add this interpretation and the illustrative example to the revised discussion.

Dear Reviewer,

Thank you for your feedback and suggestions. We fully understand you are busy with other commitments. Considering we are midway through the discussion period, could you please take a look at the rebuttal to see if we have sufficiently addressed your concerns, and if so, update the score? If not, we are happy to answer any further questions. 

Best,

Authors

Reviewer_tYzSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies if a trained policy can be recovered using synthetic data labeled by a teacher policy. The learned policy (student) only receives inputs from a fixed chosen distribution and the corresponding actions from the teacher. The authors show how query geometry affects recovery. Sampling in directions where the teacher is sensitive improves performance compared with random directions. However, if only these sensitive directions are chosen, there is a risk of missing behavior outside this subspace. The authors show these effects on four MuJoCo tasks.

Critical strengths
  • Authors show that most of the teacher's performance can be recovered using synthetic inputs with teacher labels only. Student training doesn't require usual information used for training such as expert trajectories, reward function or querying the environment.

  • Experiments help explain how query geometry affets the recovery. It shows that sampling sensitive directions is usefull, but not enough. Adding noise outside these directions also helps. The authors show similar effects with SAC and discrete action ReLU policies.

Critical weaknesses
  • Covariance experiment changes both the variance of each input dimension and the correlations between them. It is, therefore, not clear how much of the improvement comes from correlations, as claimed by the authors. Adding a comparison using only the diagonal would help to clarify this.

  • There seems to be an inconsistency in some recovery results. For example, Tables 8 and 19 for hopper, and Tables 10 and 20 for ant, report different recovery values, although both analyses describe full-width students trained with 100k samples. It is not clear what explains these differences.

Questions for authors
  • Have you tried the recovery with the diagonal of the covariance matrix alone ? How does it compare to the full covariance ?
  • Could you clarify the inconsistencies in recovery reported in table 8 and 19 and 10 and 20 ?
  • For the LunarLander, how do you compute the sensitivity matrix over discrete actions ? Do you passe them through a ReLU activation ?

Comments:

  • page 21: there is an unreferenced table (Table ??)
4 comments

We thank the reviewer for their feedback and suggestions. We address their concerns below.

W1 & Q1: Additional diagonal-variance control

Thank you for suggesting this additional clarification. We ran an additional control comparing identity, diagonal empirical covariance, and full empirical covariance at both zero and empirical means. We observe that full covariance improves recovery beyond marginal variance adaptation on both Ant and Humanoid.

For each teacher, we compare N(m,I)\mathcal N(m,I), N(m,diag⁡Σ^)\mathcal N(m,\operatorname{diag}\hat\Sigma), and N(m,Σ^)\mathcal N(m,\hat\Sigma) at m=0m=0 and m=μ^m=\hat\mu. The diagonal and full conditions use the same covariance estimate, so their marginal variances and total variance are identical. We estimate covariance from 10,000 teacher-rollout states. Each condition uses ten teachers per environment, 100k training queries, 20 epochs, and 100 evaluation episodes per policy.

The paired full-minus-diagonal recovery gains are:

EnvironmentZero mean: gain in pp [95% CI]Empirical mean: gain in pp [95% CI]
Ant+10.77 [6.22, 15.61]+6.48 [3.25, 10.18]
Humanoid+34.29 [28.63, 39.60]+32.82 [24.34, 41.52]

All four paired return comparisons remain significant after Holm correction (padj=0.007812p_{\mathrm{adj}}=0.007812). We report that full covariance also reduces action MSE on teacher-visited and student-visited states for both raw actor outputs and executed actions. Marginal scaling contributes as well, particularly on Ant. This supports the conclusion that correlation provides an additional benefit in these settings.

W2 &Q2: Recovery result corrections

We have identified the corrections in the paper. Our smoothness-table generators incorrectly pooled four sample budgets and three student widths, causing this error. The reported SEM also summarized variation across configuration means rather than seeds. We have regenerated every affected column using the correct cohort and seed-level aggregation.

The corrected gaussian recovery for hopper is 93.7% and 81.9% for ant, which match tables 8 and 10. We will update the associated SD/SEM labels and metric descriptions. These corrections have used saved training data.

Q3: LunarLander sensitivity matrix

For LunarLander, we differentiate the continuous logit vector before action selection. ℓ(x)\ell(x) denotes the actor logits and the sensitivity quantity is

\qquad G=\frac{1}{N}\sum_{i=1}^{N}J_i^\top J_i.$$ We do not differentiate the discrete selected action. ReLU is also the hidden layer activation, we do not apply it to the action indices. We will state this explicitly in the relevant section of the appendix. The unresolved reference on page 21 points to the mean/covariance decomposition in Table 25. We will correct that cross-reference.

Dear Reviewer,

Thank you for your feedback and suggestions. We fully understand you are busy with other commitments. Considering we are midway through the discussion period, could you please take a look at the rebuttal to see if we have sufficiently addressed your concern? If not, we are happy to answer any further questions. 

Best, Authors

I thank the authors for answering the questions. The new diagonal-covariance control and the clarifications address my concerns. I will keep my score.

We sincerely thank the reviewer for their insightful comments and questions. The suggestions, additional requested controls and experiments have substantially improved the paper, and we will incorporate the clarifications and additional results into the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.