- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper addresses the following problem: when a state-observing expert policy is replaced by a policy that cannot observe the full state, how can one determine which components of a state estimator’s output become unreliable? The authors quantify this as the policy-swap posterior divergence (PSPD).
The authors argue that the teacher’s action information cannot reliably indicate PSPD, and propose a criterion for determining when PSPD is zero. Specifically, for a fixed observation history, if the probability that the teacher produces the logged actions (the path weight ), averaged over the state trajectories consistent with each value of the state of interest, is equal across those values, then PSPD is zero.
Experiments show that training the predictor to fit the do posterior works better in practice than fitting the behavior posterior , even though the latter scores better on evaluation data collected by the teacher.
Critical strengths
- Novel and valuable contribution as far as I can tell. The paper proposes a theoretically principled method for identifying which state components are confounded by the teacher’s actions, which is a realistic and well-defined problem. The paper is self-consistent.
- Experiments confirm the claims. The experimental results support the authors’ theoretical conclusions.
Critical weaknesses
I have to first confess that I am not very familiar with this field and had a hard time reading this paper, so I'm probably not the best person to review it, and my understanding may be incorrect.
My main concern is the presentation of this paper. The writing made it hard for me to follow:
- The second sentence of the abstract packs the main contribution into a single convoluted clause that is difficult to parse without already knowing the result.
- Key terms (PSPD, residual selection, observational-soft, do-soft) are used in the abstract and introduction before being defined in Table 1; they should be defined before they are used.
- The symbol is overloaded without comment. It denotes the observation model in Section 3 and the deployment gap in Section 4.4 and Table 4.
Questions for authors
- The criterion requires exact equality of the averaged path weight across target values. For a non-rule-based teacher, or for large, continuous, or long-horizon state spaces where the average must be estimated, exact equality cannot be verified. How much deviation is tolerable before PSPD can no longer be considered approximately zero?
- In RockSample, the predictor trained on does not approach its own target, and the learned gap is only 13% of the exact distortion when the teacher attends to the state (). Why is harder to fit than , and how does this affect the interpretation of the RockSample results?
2 comments
Thank you for the response.
I appreciate the authors' response, and acknowledge that I have read it. The notation and presentation issues have been largely resolved. Good luck!
Approximate residual selection, fitting, and clearer presentation
Thank you for the detailed presentation comments.
1. Presentation.
The revised abstract uses shorter sentences and introduces the phenomenon before the criterion. The Introduction defines PSPD, observational-soft, do-soft, and residual selection at first use and describes the two training losses in plain words; Section 3 defines full privilege (λ=1), and Table 1 remains a reference glossary. The observation model is now O, leaving G for the deployment gap in Section 4.4 and Table 4. The teacher's action-probability ratio in Appendix F.2 is now κ, so it no longer shares a symbol with the weight ratio r of Section 4.1.
2. How much deviation from equality is tolerable?
Short answer: if the teacher's mean path weight varies by at most a relative ±δ across target values, the distortion is at most −21log(1−δ2) nats, about δ2/2 for small δ; a 10% variation gives at most 0.00503 nats per target. Formally, at a fixed history h, write p=bdo,Z(⋅∣h), b=bμ,Z(⋅∣h), and
r(z,h)=Edo[W∣h]Edo[W∣Z=z,h]=dpdb(z).
Under the paper's support assumptions, if ∣r(z,h)−1∣≤δ<1 throughout the target support, then
KL(p∥b)≤−21log(1−δ2).
A uniform bound over deployment histories gives the same bound on PSPD; a history-dependent bound is averaged over those histories. A sufficient condition for a tolerance τ is δ≤1−e−2τ.
The result also applies to continuous targets and non-rule-based teachers. For estimated ratios, if ∥r−r∥∞≤η and ∥r−1∥∞≤δ, substitute δ+η for δ when this sum is below 1; this needs uniform error control over the deployment support. Normalization keeps the condition meaningful at long horizons, where absolute path weights are tiny.
Section 4.1 states the bound; Appendix G.1 gives its proof and estimation conditions.
3. Why is observational-soft harder to fit, and how should the 13% gap be interpreted?
The two targets have different structure; Appendix C.3 now states this. Under the do law, the RockSample posterior factorizes over rocks (Appendix F, Eq. 75): each rock's marginal changes only with that rock's own check outcomes and whether it has been sampled. At full privilege, the behavior posterior is proportional to this product times the teacher's path weight C(h)κc(x) (Appendix F.2), where c(x) counts past actions the teacher would have preferred under the whole configuration x and κ≈144 is the ratio of preferred to nonpreferred action probabilities. Each such action can shift a configuration's log weight by about 5 nats, so fitting bμ requires the teacher's layout-specific preferred actions, which depend on all eleven rock qualities. The diagnostics agree (Table 15, Appendix C.3): from selected to final checkpoints, observational own-target KL falls from 0.164 to 0.012 nats per unresolved rock on training histories but rises from 0.215 to 0.705 on held-out layouts, where do-soft's KL is 0.003. More layouts lower the selected-checkpoint held-out KL to 0.158 (64 layouts of 256 episodes) and 0.131 (256 layouts; Table 20). We do not identify the optimization dynamics behind this overfitting.
In plain terms, own-target error weights outcomes by bμ, whereas the deployment penalty is largest where bμ gives little probability to values the do posterior considers likely, so a fitted predictor can be close to bμ yet realize only part of the distortion. On common deployment histories, G=Dmarg−Edo+Bobs (exact distortion, do-side error, signed observational term), with Dmarg=18.083, Edo=0.147, Bobs=−15.535 and G=2.402 nats (Table 4A). Target-history pairs with local KL of at least 1 nat carry 97.1% of Dmarg, with a mean of 5.9 nats per coordinate (Appendix G.2). For one binary target with p=0.5 and b=0.999, a prediction q=0.9 is within 0.10 nats of its own target, KL(b∥q), yet KL(p∥q)=0.51 against KL(p∥b)=2.76: it realizes 18% of the distortion. The RockSample sign holds in every seed of the main studies. Its size combines how much of bμ's action-based inference the predictor reproduces on distorted targets with fitting error that differs between the arms: at the selected eight-layout checkpoints, near-zero-distortion coordinates carry 31.8% of the gap. With 64 layouts, a comparison that also adds data and stops observational-soft later, this share falls to 12.2% and the gap in the most distorted bin rises from 0.353 to 0.639 nats per coordinate (Figure 3, Appendix G.2).
Changes in the revised manuscript are highlighted in blue.