acceptodds
Under review as a conference paper at ICLR 2027

Target-Specific Distortion in State Supervision under Policy Swap

Abstract

History encoders for partially observable control are often trained on hidden-state labels logged by a privileged teacher that sees the state, then deployed with a policy that sees only past observations and actions. The labels are correct, but the teacher's actions also reveal what it saw, so training fits a posterior that this policy swap invalidates. Prior analyses treat the hidden state as a whole. We show that the swap distorts a prediction target exactly when, at histories that deployment reaches, the teacher's action likelihood averaged over hidden paths consistent with the history varies with the target's value, and that only the part of the teacher's action information about the target can distort it, although this part need not do so. The exact posterior fitted to the log scores better on teacher histories and worse at deployment, and an exact identity splits a trained predictor's gap into distortion and fitting error. In synthetic channels with fixed histories and action information, coupling the target to the teacher's actions alone reverses the learned risk ordering in every seed, and where this condition and per-target action information disagree, trained predictors follow the condition. On the RockSample benchmark, predictors trained on exact posterior labels show the deployment gap in every seed at every tested level of teacher privilege. At full privilege, fixed controllers acting on them earn more return with deployment-posterior labels, fitting error rather than exact distortion sets the gap's size, and validation on teacher histories favored early checkpoints of the distorted predictors, but only under a loss restricted to targets the history leaves undetermined. When the environment model is known or identifiable from the state-labeled log, relabeling with the deployment posterior removes the distortion. Known teacher propensities allow only a high-variance reweighted evaluation, and otherwise the remedy is history-only data collection.

Primary area:
reinforcement learning
Keywords:
partially observable Markov decision processes, privileged information, belief learning, causal inference, policy swap, model selection

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_e99UOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the following problem: when a state-observing expert policy is replaced by a policy that cannot observe the full state, how can one determine which components of a state estimator’s output become unreliable? The authors quantify this as the policy-swap posterior divergence (PSPD).

The authors argue that the teacher’s action information cannot reliably indicate PSPD, and propose a criterion for determining when PSPD is zero. Specifically, for a fixed observation history, if the probability that the teacher produces the logged actions (the path weight WW), averaged over the state trajectories consistent with each value of the state of interest, is equal across those values, then PSPD is zero.

Experiments show that training the predictor to fit the do posterior bdob^{do} works better in practice than fitting the behavior posterior bμb^\mu, even though the latter scores better on evaluation data collected by the teacher.

Critical strengths
  • Novel and valuable contribution as far as I can tell. The paper proposes a theoretically principled method for identifying which state components are confounded by the teacher’s actions, which is a realistic and well-defined problem. The paper is self-consistent.
  • Experiments confirm the claims. The experimental results support the authors’ theoretical conclusions.
Critical weaknesses

I have to first confess that I am not very familiar with this field and had a hard time reading this paper, so I'm probably not the best person to review it, and my understanding may be incorrect.

My main concern is the presentation of this paper. The writing made it hard for me to follow:

  • The second sentence of the abstract packs the main contribution into a single convoluted clause that is difficult to parse without already knowing the result.
  • Key terms (PSPD, residual selection, observational-soft, do-soft) are used in the abstract and introduction before being defined in Table 1; they should be defined before they are used.
  • The symbol GG is overloaded without comment. It denotes the observation model in Section 3 and the deployment gap in Section 4.4 and Table 4.
Questions for authors
  • The criterion requires exact equality of the averaged path weight WW across target values. For a non-rule-based teacher, or for large, continuous, or long-horizon state spaces where the average must be estimated, exact equality cannot be verified. How much deviation is tolerable before PSPD can no longer be considered approximately zero?
  • In RockSample, the predictor trained on bμb^\mu does not approach its own target, and the learned gap is only 13% of the exact distortion when the teacher attends to the state (λ=1\lambda = 1). Why is bμb^\mu harder to fit than bdob^{do}, and how does this affect the interpretation of the RockSample results?
2 comments
AuthorsOpenReview ↗

Approximate residual selection, fitting, and clearer presentation

Thank you for the detailed presentation comments.

1. Presentation.

The revised abstract uses shorter sentences and introduces the phenomenon before the criterion. The Introduction defines PSPD, observational-soft, do-soft, and residual selection at first use and describes the two training losses in plain words; Section 3 defines full privilege (λ=1\lambda=1), and Table 1 remains a reference glossary. The observation model is now O\mathcal O, leaving GG for the deployment gap in Section 4.4 and Table 4. The teacher's action-probability ratio in Appendix F.2 is now κ\kappa, so it no longer shares a symbol with the weight ratio rr of Section 4.1.

2. How much deviation from equality is tolerable?

Short answer: if the teacher's mean path weight varies by at most a relative ±δ\pm\delta across target values, the distortion is at most −12log⁡(1−δ2)-\tfrac12\log(1-\delta^2) nats, about δ2/2\delta^2/2 for small δ\delta; a 10% variation gives at most 0.00503 nats per target. Formally, at a fixed history hh, write p=bdo,Z(⋅∣h)p=b^{\mathrm{do},Z}(\cdot\mid h), b=bμ,Z(⋅∣h)b=b^{\mu,Z}(\cdot\mid h), and

r(z,h)=Edo[W∣Z=z,h]Edo[W∣h]=dbdp(z).r(z,h)=\frac{\mathbb E_{\mathrm{do}}[W\mid Z=z,h]}{\mathbb E_{\mathrm{do}}[W\mid h]}=\frac{db}{dp}(z).

Under the paper's support assumptions, if ∣r(z,h)−1∣≤δ<1|r(z,h)-1|\leq\delta<1 throughout the target support, then

KL(p∥b)≤−12log⁡(1−δ2).\mathrm{KL}(p\Vert b)\leq-\tfrac12\log(1-\delta^2).

A uniform bound over deployment histories gives the same bound on PSPD; a history-dependent bound is averaged over those histories. A sufficient condition for a tolerance τ\tau is δ≤1−e−2τ\delta\leq\sqrt{1-e^{-2\tau}}.

The result also applies to continuous targets and non-rule-based teachers. For estimated ratios, if ∥r^−r∥∞≤η\|\widehat r-r\|_\infty\leq\eta and ∥r^−1∥∞≤δ^\|\widehat r-1\|_\infty\leq\widehat\delta, substitute δ^+η\widehat\delta+\eta for δ\delta when this sum is below 1; this needs uniform error control over the deployment support. Normalization keeps the condition meaningful at long horizons, where absolute path weights are tiny.

Section 4.1 states the bound; Appendix G.1 gives its proof and estimation conditions.

3. Why is observational-soft harder to fit, and how should the 13% gap be interpreted?

The two targets have different structure; Appendix C.3 now states this. Under the do law, the RockSample posterior factorizes over rocks (Appendix F, Eq. 75): each rock's marginal changes only with that rock's own check outcomes and whether it has been sampled. At full privilege, the behavior posterior is proportional to this product times the teacher's path weight C(h) κc(x)C(h)\,\kappa^{c(x)} (Appendix F.2), where c(x)c(x) counts past actions the teacher would have preferred under the whole configuration xx and κ≈144\kappa\approx144 is the ratio of preferred to nonpreferred action probabilities. Each such action can shift a configuration's log weight by about 5 nats, so fitting bμb^\mu requires the teacher's layout-specific preferred actions, which depend on all eleven rock qualities. The diagnostics agree (Table 15, Appendix C.3): from selected to final checkpoints, observational own-target KL falls from 0.164 to 0.012 nats per unresolved rock on training histories but rises from 0.215 to 0.705 on held-out layouts, where do-soft's KL is 0.003. More layouts lower the selected-checkpoint held-out KL to 0.158 (64 layouts of 256 episodes) and 0.131 (256 layouts; Table 20). We do not identify the optimization dynamics behind this overfitting.

In plain terms, own-target error weights outcomes by bμb^\mu, whereas the deployment penalty is largest where bμb^\mu gives little probability to values the do posterior considers likely, so a fitted predictor can be close to bμb^\mu yet realize only part of the distortion. On common deployment histories, G=Dmarg−Edo+BobsG=D_{\mathrm{marg}}-E_{\mathrm{do}}+B_{\mathrm{obs}} (exact distortion, do-side error, signed observational term), with Dmarg=18.083D_{\mathrm{marg}}=18.083, Edo=0.147E_{\mathrm{do}}=0.147, Bobs=−15.535B_{\mathrm{obs}}=-15.535 and G=2.402G=2.402 nats (Table 4A). Target-history pairs with local KL of at least 1 nat carry 97.1% of DmargD_{\mathrm{marg}}, with a mean of 5.9 nats per coordinate (Appendix G.2). For one binary target with p=0.5p=0.5 and b=0.999b=0.999, a prediction q=0.9q=0.9 is within 0.10 nats of its own target, KL(b∥q)\mathrm{KL}(b\Vert q), yet KL(p∥q)=0.51\mathrm{KL}(p\Vert q)=0.51 against KL(p∥b)=2.76\mathrm{KL}(p\Vert b)=2.76: it realizes 18% of the distortion. The RockSample sign holds in every seed of the main studies. Its size combines how much of bμb^\mu's action-based inference the predictor reproduces on distorted targets with fitting error that differs between the arms: at the selected eight-layout checkpoints, near-zero-distortion coordinates carry 31.8% of the gap. With 64 layouts, a comparison that also adds data and stops observational-soft later, this share falls to 12.2% and the gap in the most distorted bin rises from 0.353 to 0.639 nats per coordinate (Figure 3, Appendix G.2).

Changes in the revised manuscript are highlighted in blue.

Reviewer_e99UOpenReview ↗

Thank you for the response.

I appreciate the authors' response, and acknowledge that I have read it. The notation and presentation issues have been largely resolved. Good luck!

Reviewer_a5GvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper examines an easily overlooked issue in state supervision: training on data collected by a privileged teacher can produce misleading deployment predictions even when all state labels are correct. A predictor may exploit hidden-state information encoded in the teacher’s actions. Once the teacher is replaced by a policy that uses only public history, those actions no longer provide the same additional evidence, although their effects on the environment remain relevant. The authors characterize which prediction targets are affected through a residual-selection criterion, showing that the same teacher data can distort some target posteriors while leaving others unchanged. Synthetic experiments and RockSample studies demonstrate the phenomenon and show that supervision using interventional (do) posterior labels can improve deployment prediction. The central message is that correct labels alone do not guarantee valid deployment inferences: the information available to the data-collection policy also matters.

Critical strengths
  1. The target-level analysis is clear and insightful. Under its stated assumptions, the paper provides a necessary and sufficient condition for target-specific posterior distortion after a policy swap. Its decomposition of action information explains why more hidden-state information in the teacher’s actions does not necessarily imply greater prediction distortion. This makes the analysis specific to individual state variables and provides a theoretical basis for assessing which supervision targets remain valid.

  2. The controlled experiments and error analysis are convincing. The matched comparisons hold histories, architecture, initialization, and training procedures fixed while changing only the supervision labels, providing direct evidence of better prediction on teacher data but worse prediction after deployment. The analyses of fitting error and model selection clarify the scope of these findings. The supplementary code and experimental records also facilitate checking the reported results.

Critical weaknesses
  1. The connection between theoretical distortion and learned performance remains insufficiently direct in RockSample. The observational-soft models do not adequately fit their own supervision targets, and fitting effects substantially influence the results. For example, Table 4 reports an exact aggregate distortion of approximately 18.08 nats but a deployment loss gap of approximately 2.40 nats for the selected models. Although the paper decomposes this gap, a more direct target-level analysis would help establish how informative theoretical distortion is about deployment errors in finite learned predictors.

  2. The proposed corrections depend on substantial environment knowledge. Do-posterior relabeling requires known or reliably identifiable dynamics and observation models, while target certification relies on specific structural assumptions. The main evidence comes from environments permitting exact posterior computation; high-dimensional visual observations, continuous actions, and neural teachers remain untested. The practical applicability of these corrections beyond such tractable settings, including to real-world robotics, therefore remains uncertain.

Questions for authors

To what extent does target-specific distortion explain deployment performance in finite learned models?

In fully privileged RockSample, the observational-soft models do not adequately fit their own targets. Table 4 reports an exact aggregate distortion of approximately 18.08 nats but a deployment loss gap of approximately 2.40 nats for the selected models. While the paper already decomposes the aggregate gap, could the authors provide a corresponding target-level analysis using existing checkpoints or training configurations?

Specifically, do targets with theoretical distortion also exhibit larger deployment loss gaps between observational-soft and do-soft predictors? Relating each target’s exact distortion, fitting errors—including the signed fitting term in Eq. (11)—and deployment loss gap, and examining whether this relationship becomes clearer with improved fitting, would help assess the explanatory value of the target-level analysis for learned models.

2 comments
AuthorsOpenReview ↗

Target-level distortion, fitting error, and practical scope

Thank you for this precise question. In short: at the submitted selected checkpoints, exact distortion and learned gaps are only moderately associated across output coordinates, and fitting terms set the size of the target-level gaps. With 64 training layouts and lower own-target error, the distortion strata separate more clearly at selected checkpoints; at final checkpoints both conditions realize far more distortion (Appendix G.2). We only rescored existing checkpoints.

1. Target-level learned performance.

We use all five seeds, selected and final checkpoints, and the histories and weights of Table 4A. The new tables report Dj,Eobs,j,Edo,j,Bobs,j,GjD_j,E_{\mathrm{obs},j},E_{\mathrm{do},j},B_{\mathrm{obs},j},G_j for every output coordinate, with paired-seed intervals for GjG_j. We compute the signed term directly from log probabilities and check

Gj=Dj−Edo,j+Bobs,j.G_j=D_j-E_{\mathrm{do},j}+B_{\mathrm{obs},j}.

For the selected eight-layout models, the prespecified Pearson correlation across eleven output coordinates is 0.719 on loss masses ([0.669, 0.751]; intervals reflect training seeds with outputs fixed). Rocks unresolved more often accumulate more of both, so we also normalize per unresolved coordinate (post hoc): 0.506 [0.452, 0.541]. Because output indices change spatial meaning across layouts, we also bin target-history pairs by exact local KL (bins fixed before rescoring):

  • The near-zero-distortion bin (local KL below 10−610^{-6}; 38.1% of unresolved coordinates) has a gap of 0.239 [0.231, 0.248] nats per coordinate, from the arms' different fitting errors; it carries 31.8% of the aggregate gap (a post hoc share).
  • In the high-distortion bin (local KL at least 1 nat), mean D=5.883D=5.883, Bobs=−5.516B_{\mathrm{obs}}=-5.516 and Edo=0.013E_{\mathrm{do}}=0.013 give G=0.353G=0.353 [0.339, 0.372] (rounded terms). This bin holds 97.1% of DD but only 43.9% of the gap. The intermediate-bin gaps are nonmonotone.

The signed term accounts for the compressed gap at high distortion and the positive gap on undistorted targets (Figure 3, Section 5.5, Appendix G.2).

2. Improved fitting.

On the same histories, the existing 64-layout models have lower selected-model own-target error: EobsE_{\mathrm{obs}} falls from 2.504 to 1.465 nats (-1.039 [-1.064, -1.011], all five seeds). The near-zero-bin gap falls from 0.239 to 0.096 and its share of the gap from 31.8% to 12.2%; the high-distortion-bin gap rises from 0.353 to 0.639 and its share from 43.9% to 75.7%. The bin means now increase with distortion, though only the top bin separates clearly, and the per-coordinate correlation rises from 0.506 to 0.744 (+0.238 [0.154, 0.318], positive in every seed). The aggregate gap changes from 2.402 to 2.520 nats (0.118 [-0.178, 0.476]); the lower EdoE_{\mathrm{do}} contributes 0.055.

This comparison does not isolate fit. More episodes on the original eight layouts left observational-soft's fit and G/DG/D unchanged, while more layouts improved it (Table 20); the same extra data alone lowers do-soft's EdoE_{\mathrm{do}}. Observational-soft selection also stops later (2,000-3,500 versus 250 updates), and longer training enlarges the gap in every bin: at final checkpoints the highest-bin gaps are 2.745 and 1.870 for eight and 64 layouts, and the near-zero-bin gaps 0.518 and 0.202. There, the better-fitting 64-layout models track DD slightly less closely (per-coordinate correlation 0.836 versus 0.905). Own-target error averages outcomes under bμb^\mu, whereas BobsB_{\mathrm{obs}} averages them under bdob^{\mathrm{do}}, so reducing one need not remove the other.

3. Environment knowledge and practical scope.

Do relabeling uses initial, transition, and observation models that are known or identifiable on the required support from state-labeled logs; identification needs consistency, invariant factors, coverage and recoverability, not teacher propensities (Appendix A.9). The general criterion also needs teacher action likelihoods. Within its structural assumptions (Section 6), the RockSample certificate stayed exact at full privilege under every misspecified analyst model we tried (at λ=0.5\lambda=0.5 only with the logged student map), whereas a 10−410^{-4} action-information threshold and a plug-in KL cleared targets with true KL up to 16.3 and 24.4 nats (Section 5.7, Appendix C.8).

When models are unavailable and new collection is possible, a history-only policy removes the mechanism; RMA, locomotion students and the DAgger stage of Learning by Cheating already train on their own rollouts (Appendix D.5). The risk remains wherever training histories come from a policy that acts on hidden state, including fixed offline driving logs and students that keep privileged action dependence. Validating predictions on deployment histories requires target labels or a validated posterior reference. High-dimensional observations, continuous actions, neural teachers and real robots remain untested.

Changes in the revised manuscript are highlighted in blue.

Thank you for the detailed response and the new target-level analysis, which answers my question directly.

The results show that the criterion identifies which targets are at risk but predicts the size of the learned gap only weakly. At the selected checkpoints, the most distorted bin holds 97.1% of the exact distortion and 43.9% of the gap, while near-zero-distortion coordinates still contribute 31.8%. The separation becomes clearer with 64 layouts, although this comparison does not isolate fit, and the aggregate gap remains about 13 to 14% of the exact distortion.

On practical scope, the evidence still comes from environments with exact posteriors, and the certificate has not yet been shown to improve deployment beyond full do relabeling.

I appreciate the careful and transparent revision and maintain my rating of 3

Reviewer_6kpzOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This is a well written paper about privileged teachers that presents nice theoretical analysis on the subject and good simulation results.

Sadly, it falls a little bit outside my level of expertise to be able to point to weaknesses in this paper.

Besides this, I found the paper actually clear, well-written and convincing.

Critical strengths

This paper is very well written and provides in my humble opinion convincing results.

Critical weaknesses

No specific weaknesses expect perhaps that I would have loved more the authors elaborate on the practical implications of their work.

Questions for authors

No specific questions.

1 comment
AuthorsOpenReview ↗

Practical implications of privileged state supervision

Thank you for the suggestion. Correct state labels can train a predictor to use evidence in teacher actions that disappears after a policy swap. Section 5.7 now explains the implications for validation, relabeling, and data collection:

  1. Validate on deployment histories. A held-out set collected by the same privileged teacher can reward the same action-based evidence as the training set. Evaluation on histories collected by a history-only policy tests whether predictions remain reliable after the swap, provided target labels or a validated posterior reference are available.
  2. Relabel when an environment model is available. If the initial, transition, and observation models are known, do-posterior labels retain the actions' physical effects while removing the evidence in the teacher's choices. These models can also be identified from state-labeled logs under the paper's coverage and model assumptions. Do relabeling does not require the teacher's action probabilities.
  3. When new collection is possible, use a history-only policy. Executing such a policy while retaining state labels removes the privileged-action selection mechanism. For an existing teacher log, the target-specific criterion assesses which targets remain valid under the stated assumptions.

Changes in the revised manuscript are highlighted in blue.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of author response and manuscript revisions

The revision adds the requested target-level analysis and a quantitative tolerance for approximate residual selection, and clarifies the presentation and practical implications.

  1. Target-level learned errors (Figure 3; Section 5.5; Appendix G.2). We rescore the existing 8-layout and 64-layout RockSample models on the same deployment histories, reporting exact distortion, both own-target errors, the signed observational fitting term, and the learned gap for each target and for fixed distortion bins. At the submitted selected checkpoints, fitting terms set the size of the target-level gaps: the most distorted bin holds 97.1% of exact distortion but only 43.9% of the learned gap, and near-zero-distortion coordinates contribute 31.8%. With more layouts and lower own-target error, these shares become 75.7% and 12.2% and the most distorted bin separates clearly from the rest; this comparison also adds data and stops observational-soft later, and at final checkpoints both conditions realize far more distortion (Appendix G.2). The negative signed term accounts for nearly all of the difference between exact distortion and the aggregate gap, and Appendix C.3 notes a structural difference between the two targets that is consistent with the harder fit. The analysis uses all five seeds and both selected and final checkpoints, with no new training or data collection.
  2. Approximate residual selection (Section 4.1; Appendix G.1). A normalized residual ratio satisfying ∣rh−1∣≤δ<1|r_h-1|\leq\delta<1 gives KL≤−12log⁡(1−δ2)\mathrm{KL}\leq-\tfrac12\log(1-\delta^2). We give the proof and the deployment-history coverage and estimation-error conditions.
  3. Presentation and practical implications. The abstract uses shorter sentences, and the terms Reviewer e99U listed are defined at first use; the Introduction describes the training losses in plain words, and Section 3 defines full privilege. The observation-model symbol O\mathcal O is distinct from the deployment gap GG, and the teacher's action-probability ratio is now κ\kappa. Section 5.7 explains the practical implications and the information required for each remedy.

Changes in the revised manuscript are highlighted in blue. The responses below each review give the findings and manuscript locations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.