acceptodds
Under review as a conference paper at ICLR 2027

Theoretical Insights into Informed Value Estimation for Asymmetric Recurrent Natural Actor-Critic

Abstract

In partially observable environments, reinforcement learning agents must act based on incomplete or noisy observations. While asymmetric actor-critic methods empirically improve learning, their theoretical properties remain largely unexplored. We generalize the Recurrent Natural Actor-Critic (Rec-NAC) algorithm to the asymmetric setting, where a state-dependent privileged signal is available only during training and is used by the critic. Both actor and critic consist of a recurrent encoder of the observable history followed by a linear head; the critic's head is learnable and integrates the privileged signal, while the actor does not. By adapting the finite-time and finite-width analysis of policy evaluation via recurrent temporal-difference learning to this setting, we identify distinct information and representation benefits of an informed critic and uncover a fundamental trade-off: privileged information reduces conditional target variance and removes the irreducible error due to information aliasing, while the informed architecture can improve best-in-class approximation but also induces additional finite-time complexity. We further analyze the resulting recurrent natural policy-gradient update and show that privileged signals induce an information mismatch in the actor's compatible function regression. Our results characterize when privileged information improves value estimation and its implications for policy learning.

Primary area:
reinforcement learning
Keywords:
Asymmetric RL, Privileged Signals, Actor-Critic, Partial Observability, POMDP

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jUbAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper extends Recurrent Natural Actor-Critic to an asymmetric setting in which the critic receives state-dependent privileged information while the actor observes only interaction histories. It presents finite-time and finite-width bounds for recurrent TD learning, separates information and representation effects, and analyzes how informed value estimation enters the actor’s compatible function approximation error.

Critical strengths
  1. Useful separation of information, representation, and complexity effects.

The paper distinguishes genuinely additional information from a more accessible representation of information already contained in the history. This provides a useful framework for understanding why privileged inputs need not yield uniform improvements under finite computational and statistical budgets.

  1. Explicit connection between critic estimation and actor-side approximation.

The analysis separately tracks optimization, finite-width, and truncation contributions, and connects critic estimation error to compatible regression error. This organization makes the assumptions and potential bottlenecks more explicit than a purely asymptotic convergence statement.

Critical weaknesses
  1. The Bellman relation is not justified for arbitrary state-dependent privileged signals.

Section 2.3 and Proposition E.3 assert

\mid h_t,i_t,a_t \right]$$ However, the next critic input retains only the current privileged signal $i_{t+1}$, not the previous signal $i_t$. The stated information-generation model does not guarantee that $i_t$ becomes irrelevant to future returns after conditioning on $(h_{t+1},i_{t+1},a_{t+1})$. In particular, writing $G_{t+1}$ for the remaining discounted return, the assumptions do not establish $$\mathbb E[G_{t+1}\mid h_{t+1},i_{t+1},a_{t+1},i_t]=\mathbb E[G_{t+1}\mid h_{t+1},i_{t+1},a_{t+1}].$$ For a persistent hidden state observed through independently noisy privileged measurements, the discarded measurement can still contain predictive information. Consequently, the tower property alone does not justify the proposed recursion. This affects the central convergence claim: Eq. (13) in the proof of Theorem 5.1 explicitly uses this Bellman relation. An additional sufficiency assumption, or a formulation retaining the privileged history, appears necessary before the TD fixed point and convergence analysis can be established in the claimed generality. 2. **The proposed architecture removes privileged dependence from the advantage, weakening the interpretation of actor-side information mismatch.** The critic in Section 3.2 is additive: $$\hat Q_t(h,i,a)=g_t(h,a)+\beta^\top i.$$ Applying the advantage definition in Section 4.2 therefore gives $$\hat A_t(h,i,a)=g_t(h,a)-\sum_{a'}\pi(a'\mid h)g_t(h,a'),$$ because the privileged term cancels exactly. Thus, for fixed critic parameters, the actual compatible-regression target does not depend on i. Privileged information can still affect the learned history branch indirectly through TD training; this cancellation does not make the algorithm equivalent to symmetric training. Nevertheless, it creates a substantial disconnect between the implemented architecture and the explanation that the actor must fit privileged-dependent advantage variation. Under exact realizability in the corresponding additive class, the same cancellation holds for the true advantage. Moreover, the proposed $\varepsilon_{\mathrm{info}}$ measures conditional variation in $Q^\pi$, rather than in the advantage, and can therefore count action-independent baseline variation that is irrelevant to compatible regression. The analysis should distinguish this potentially loose upper-bound contribution from genuine actor-side approximation difficulty.
Questions for authors

Why are no numerical experiments included? Could small, controlled POMDP experiments test the predicted information–complexity trade-off and whether improved critic accuracy translates into better policy performance?

4 comments

We thank the reviewer for their valuable feedback and for highlighting the key contributions of our work. In response to the review, we have revised the manuscript to address the concerns raised. For ease of reference, all significant changes in the revised manuscript are highlighted in blue. Below, we address each concern individually.

W1: Bellman relation in Proposition E.3. We thank the reviewer for catching this issue and confirm that the observation is correct for the original statement: the tower property alone does not justify the recursion when the next critic input retains only it+1i_{t+1} and not iti_t. The requirement E[Gt+1∣ht+1,it+1,at+1,it]=E[Gt+1∣ht+1,it+1,at+1]\mathbb{E}[G_{t+1} \vert h_{t+1}, i_{t+1}, a_{t+1}, i_t] = \mathbb{E}[G_{t+1} \vert h_{t+1}, i_{t+1}, a_{t+1}] is precisely the assumption that was needed, but not made explicit, in the original submission. We have fixed this issue in the revised manuscript by making this requirement explicit (Assumption 3 in Section 2.3): past privileged signals must be redundant for future returns once (ht+1,it+1,at+1)(h_{t+1}, i_{t+1}, a_{t+1}) is observed. Proposition E.3, and with it Eq. (13) in the proof of Theorem 5.1 and all downstream convergence results, are now stated and proven under this assumption; the results hold as originally claimed within this scope. Importantly, Assumption 3 is satisfied by the signal classes used in the vast majority of asymmetric actor-critic applications (full state, static latents, self-refining signals, and return-sufficient abstractions such as ϕ(st)\phi(s_t)). For a detailed discussion and examples of common privileged signals satisfying Assumption 3, we refer to our response to W1/Q1 of Reviewer XD5L.

Regarding the alternative fix suggested by the reviewer: retaining the privileged history as critic input indeed makes the sequence self-refining, so that the recursion holds without any assumption. However, the resulting input i0:ti_{0:t} grows with time tt and cannot be accommodated by the fixed-size linear readout of the proposed architecture; in practice, the sequence of past privileged signals must be compressed into a fixed-sized representation before being fed to the critic (cf. the discussion after Assumption 3 in the revised manuscript). Moreover, the particular choice of critic architecture made in this work allows us to eliminate the actor-side advantage-level information-mismatch error, as now made clear in Lemma E.2, Lemma E.4, and Corollary E.4 in Appendix E.6. We discuss this point in detail in our response to W2. We view critic architectures that can natively handle growing privileged inputs, e.g., via a recurrent privileged branch, as future work.

W2: Privileged dependence cancels in the advantage estimate. We thank the reviewer for this observation, and indeed the privileged block cancels in the estimate of the informed advantage function: A^t(h,i,a)=gt(h,a)−∑a′π(a′∣h)gt(h,a)\hat{A}_t(h,i,a) = g_t(h,a) - \sum_{a^\prime} \pi(a^\prime \vert h) g_t(h,a) is ii-independent for every fixed critic parameter. Privileged information thus reaches the compatible regression only indirectly, through the TD training of the recurrent representation ftf_t; we agree that this does not make the algorithm equivalent to symmetric training.

We view this cancellation as a positive structural property of the proposed architecture, which yields the best of both worlds: on the critic side, privileged information reduces the variance of the TD target (Proposition E.1) and removes the information-aliasing error (Lemma 5.1); on the actor side, the same privileged variation does not enter the compatible-regression target as additional noise. In particular, the second moment of the informed advantage target is not affected by additional variation due to the privileged signal, and the actor-side difficulty reduces to the actor’s expressive capacity for the history-measurable target, evaluated at the TD-learned history representation of the informed critic, together with εtd,n\varepsilon_{\text{td}, n} and εsgd,n\varepsilon_{\text{sgd}, n}. Intuitively, the critic exploits what the actor cannot see, while the actor's learning problem is not burdened by variation it could never fit.

We have made the following changes to the manuscript to account for this finding (marked in blue in the revision): Remark 4.1 (proof in Lemma E.2, Appendix E.6) states the ii-independence of the advantage estimate; Lemmas E.3-E.4 prove the baseline invariance and the orthogonality of the privileged value deviation; Corollary E.4 shows that the advantage-level floor vanishes under realizability; Remark E.1 identifies the QQ-based error with the information-aliasing error; and Section 5.3 states explicitly that εinfo,n\varepsilon_{\text{info}, n} should be read as a conservative surrogate, distinct from genuine actor-side approximation difficulty.

Regarding the QQ-based vs. advantage-based target (the former mirrors the definition for the symmetric setting in Cayci and Eryilmaz (2025)): the reviewer is correct that εinfo,n\varepsilon_{\text{info}, n} being defined w.r.t. QπQ^\pi can include action-independent baseline variation irrelevant to compatible regression. This looseness is not harmful. By the baseline invariance of compatible function regression, the QQ-based and advantage-based objectives differ by a qq-independent constant, without biasing the update direction. Formally, εinfo,n=εinfo,nA+∑t<TγtE[(Vtπ(ht,it)−Eit∣ht[Vtπ(ht,it)])2]\varepsilon_{\text{info}, n} = \varepsilon_{\text{info}, n}^A + \sum_{t<T}\gamma^t \mathbb{E}[(V^\pi_t(h_t, i_t) - \mathbb{E}_{i_t \vert h_t}[V^\pi_t(h_t, i_t)])^2], where the advantage-level information-mismatch error εinfo,nA\varepsilon_{\text{info}, n}^A isolates the genuine action-dependent privileged variation and the second term cancels in the advantage. Under realizability in the additive class (Assumption 4 in the revised manuscript), the true informed advantage is ii-independent almost surely, so εinfo,nA=0\varepsilon_{\text{info}, n}^A = 0, and Remark E.1 gives gives the exact identity εinfo,n=∑t<Tγtεalias,t\varepsilon_{\text{info}, n} = \sum_{t<T}\gamma^t \varepsilon_{\text{alias}, t}; the conditional variance removed on the critic side is counted on the actor side instead, without changing the regression minimizer. The advantage-level error term, εinfo,nA\varepsilon_{\text{info}, n}^A becomes non-zero only in the non-realizable regime or for critics with action-dependent privileged coupling; we leave the analysis of latter to future work. For a more detailed discussion, we refer to Section 5.3 in the revised manuscript.

Q1: Lack of numerical experiments. This point overlaps with the concern raised in W2 by Reviewer XD5L; we refer to our response there. In brief: we focused this work on a clean theoretical treatment, and we agree that small, controlled POMDP experiments testing the described information–complexity trade-off are well aligned with the validation our theory invites. We propose to add experiments on a small T-maze POMDP in which a cue is presented only once and must be remembered.

We hope these answers fully settle all open questions. Please do not hesitate to ask us to elaborate further if anything remains unclear.

Thanks for the detailed responses and the additional experiments, which address most of my concerns.

However, I believe the manuscript still needs some clarification and revision:

Scope of the theoretical claims: The new assumption 3 substantially restricts the class of admissible privileged signals. However, the manuscript still describes the setting such as arbitrary state-dependent privileged signals.

Policy-level validation: The new experiments mainly validate critic estimation and the proposed error decompositions. It would be helpful to clarify that these results do not yet establish improvements in end-to-end policy learning or final policy performance.

We thank the reviewer once again for their valuable feedback, and for listing their remaining concerns. Below, we address these concerns.

Scope of the theoretical claims.

We agree that Assumption 3 restricts the class of admissible state-dependent privileged signals and that describing our setting as allowing arbitrary state-dependent privileged signals is therefore too broad. This imprecise wording carried over from the original submission. We have now revised the corresponding statements throughout the manuscript to clarify that our analysis concerns state-dependent privileged signals satisfying Assumption 3. In particular, we have made this explicit in the Introduction (Section 1), the analysis (Section 5), and the Conclusion (Section 6).

Policy-level validation.

We agree that the experiments presented in our response to Reviewer XD5L primarily validate the critic-side theoretical results (variance reduction, elimination of information-aliasing error, and additional complexity costs) and the actor-side approximation error decomposition; since these are the main results of our theoretical analysis. As the reviewer correctly notes, they do not establish improvements in end-to-end policy learning or final policy performance. We also do not make such claims for asymmetric Rec-NAC in the manuscript. In fact, we explicitly state that improvements in value estimation do not automatically translate into uniformly tighter policy-learning guarantees when the critic has access to information unavailable to the actor (cf. Section 6).

To complement our empirical results, we have extended our experiments to the full Rec-NAC learning cycle, including policy optimization. The following table reports preliminary results on a corridor of length L=8L=8 with p=0.1p=0.1 and κ=1.0\kappa=1.0 (N=500N=500, Ktd=1000K_{\text{td}}=1000, Ksgd=2000K_{\text{sgd}}=2000, mF=mG=64m_F=m_G=64), averaged over ten independent runs (mean ±\pm standard deviation). AUC denotes the normalized area under the mean empirical episodic return throughout training. Final return, success rate, and critic's MSE are averaged over the last five policy iterations. it=∅i_t=\emptyset corresponds to the symmetric setting.

iti_tAUC (mean return)final returnsuccess ratecritic's MSE
∅\emptyset0.381 ±\pm 0.0600.447 ±\pm 0.0770.722 ±\pm 0.0375.171 ±\pm 0.042
(t,z)(t, z)0.389 ±\pm 0.0710.458 ±\pm 0.0840.731 ±\pm 0.0414.614 ±\pm 0.161
γL−tz\gamma^{L-t} z0.389 ±\pm 0.0610.469 ±\pm 0.0780.731 ±\pm 0.0362.641 ±\pm 0.389

The informed critic achieves comparable policy performance to the symmetric critic. Although mean performance is slightly higher for the asymmetric variants with access to the full-state and abstraction signals, the differences are not substantial. These findings are consistent with our theoretical analysis. For the proposed critic architecture, the privileged branch cancels in the advantage estimate, so privileged information does not directly alter the actor's regression target. Instead, it can affect policy learning indirectly through improvements in the critic's history representation.

To conclude, our theoretical analysis characterizes how critic and advantage-estimation errors enter the policy update, but does not establish that improved value estimation necessarily translates into better policy performance, particularly under limited training budgets. We will make this explicit in the camera-ready revision.

We hope that these clarifications and additional results address your remaining concerns.

Reviewer_xoRkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends the Recurrent Natural Actor-Critic (Rec-NAC) algorithm in standard POMDP to the informed POMDP setting, where the critic becomes asymmetric in the sense that it can access extra information during training. Built on the analysis by Cayci&Eryilmaz (2025), this paper establishes similar finite-time bounds for the asymmetric critic, and quantitively analyse the effect of introducing the asymmetric structure to the critic and the actor.

Critical strengths

The theoretical analysis is mathematically solid. The intuitive explanations of the theoretical results are detailed.

Critical weaknesses
  1. The technical contribution is limited. The proposed algorithmic framework is a simple combination of the Informed Asymmetirc Actor-Critic (Ebi et al., 2026) and Recurrent Natural Actor-Critic (Cayci & Eryilmaz, 2026). The theoretical analysis of asymmetric Rec-TD is a natural extension of the symmetric version (Cayci & Eryilmaz, 2026).
  2. Some theoretical understandings are mathematically correct, but not very informative. The error decomposition results (Lemma 5.1 and Prop. E.2) and the following definitions of ϵalias\epsilon_{alias} and ϵinfo\epsilon_{info} seem to be redundant. Prop. 5.1 is just simple calculation, and it has very limited implications to pracitcal use.
  3. Some notations are inconsistent. The pesudo-code of Algorithm 1 in Appendix C seems problematic. kk and KsgdK_{sgd} in Section 4.2 are not defined.
Questions for authors
  1. What is the main challenge of extending Rec-NAC to asymmetric settings?
  2. What is the implications of your theoretical analysis to practical implementations of the algorithm?
3 comments

We thank the reviewer for their valuable feedback. We appreciated the recognition of our presentation clarity around our theoretical results. Below, we address the concerns.

W1: Incremental technical contribution. We agree that our finite-time analysis builds on the analysis of symmetric Rec-NAC. This is intentional: the goal is to isolate the theoretical consequences of introducing privileged information, not to propose a new optimization algorithm. However, moving from the symmetric to the asymmetric setting is not a trivial plug-in step. The asymmetric critic changes the mathematical objects appearing in the analysis in different ways:

  • NTK and hypothesis class. The informed critic augments the recurrent representations with privileged inputs, leading to a different NTK feature map and RKHS. This changes both the class of functions that the critic can represent and how large the critic’s hypothesis class is.
  • Distributional effects. Conditioning on privileged information changes the conditional distribution of the TD targets. By Proposition E.1, the informed TD target is an unbiased estimator of the symmetric history-based QQ-function with strictly smaller conditional variance whenever the privileged signal resolves return-relevant uncertainty beyond the history-action pair. This variance reduction is a property of the information structure, that is, conditioning on a richer σ\sigma-algebra, and holds independently of the specific optimization algorithm.
  • Irreducible error term. Lemma 5.1 decompose the symmetric critic’s error into a standard regression term plus an information-aliasing term that arises because a history-only critic cannot distinguish latent states that look identical from (ht,at)(h_t, a_t) alone. This aliasing term induces a strictly positive, trajectory-level error floor. It is implicitly present in the symmetric Rec-NAC bound but was not isolated. Our asymmetric analysis makes this component explicit and shows that the informed critic’s bound removes exactly this aliasing term. We view this irreducible error decomposition as a central contribution of our work.

We also note, more technically, that the asymmetric Bellman recursion itself requires an explicit sufficiency condition on the privileged signal (Assumption 3, added in the revision; see our response to W1/Q1 of Reviewer XD5L), and that the additive readout design guarantees that the actor-side advantage floor vanishes (Corollary E.4 in the revised manuscript).

Thus, while our proof technique inherits projected semi-gradient TD and NTK machinery from Rec-NAC, the main technical contribution lies in analyzing and characterizing how leveraging privileged information affects convergence, through reduced TD-target variance and improved approximation error.

W2: Results are mathematically correct but not very informative. We respectfully argue that the informativeness of these results lies in what they isolate, not in the difficulty of the calculations:

  • Lemma 5.1: the decomposition shows that the aliasing term is irreducible under the actor's information constraint. Notably, no symmetric critic, regardless of its capacity, can remove it. This is a statement about the information structure, not the function class, and it is what licenses the subsequent comparison with the informed critic. εalias\varepsilon_{\text{alias}} and εinfo\varepsilon_{\text{info}} are not redundant: the former is the critic-side floor of the symmetric critic w.r.t. the informed target, the latter the actor-side floor of the compatible regression. That they coincide in the realizable additive case (Remark E.1) is itself informative, as it shows that the critic-side aliasing reappears on the actor side in the QQ-based formalization of the error floor.
  • Proposition 5.1: We agree that it is a simple calculation and not a directly computable selection rule (the manuscript says so). Its role is to identify which quantities govern whether privileged information pays off (the accumulated aliasing reduction vs. the complexity gap). This yields concrete design guidance (see our response to Q2 below).

W3: Notation inconsistencies. We thank the reviewer for pointing these out and have made the following corrections:

  • KsgdK_{\text{sgd}}, the number of SGD steps per policy update cycle, is now defined at first use in Section 4.2 (together with KtdK_{\text{td}}).
  • Algorithm 1 and 2 has been revised so that all indices, step sizes, and symbols are defined and consistent with Eq. (4)–(5).

Q1: Main challenge of extending Rec-NAC to asymmetric settings. The main challenge is that privileged information changes the statistical learning problem, not just the critic's input: the informed critic targets a different function class (different NTK and RKHS), estimates targets with different conditional distributions (reduced variance, Proposition E.1), and faces an approximation landscape with an irreducible information-aliasing floor that has no counterpart in the symmetric analysis (Lemma 5.1). A further, more technical challenge is that the actor-side analysis must remain consistent with the actor's information constraint: the compatible regression requires history-measurable targets, and the architecture must be designed so that privileged variation does not harm the actor's regression (Remark 4.1, Corollary E.4). Otherwise, we face additional error terms in the compatible-function approximation.

Q2: Implications for practical implementations. Our analysis yields four concrete guidelines:

  • Choose signals that resolve return-relevant uncertainty left after conditioning on (ht,at)(h_t, a_t), quantified by the conditional-variance reduction (Proposition E.1) and the aliasing error (Lemma 5.1).
  • Integrate the signal additively and action-independently in the critic readout, which guarantees the actor-side mismatch vanishes, whereas action-dependent privileged coupling reintroduces a residual error.
  • Expect net gains in partially observable domains with substantial latent-state aliasing. In fully observable settings, the aliasing benefit vanishes and privileged information only adds complexity cost (Proposition 5.1).
  • The training procedure itself is unchanged. Privileged information enters only through the critic input, so it can be adopted in an existing Rec-NAC implementation without modifying the optimization scheme.

We hope that these clarifications will have answered your concerns. Please do not hesitate to request additional explanations.

Thank you for your response. However, my concerns on the novelty and significance of this paper still remains. I will keep my score.

Reviewer_XD5LOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors study recurrent natural actor-critic learning in an asymmetric setting, when the critic, but no the actor, have access to extra information during training. They propose a recurrent critic with a trainable linear head, bound the error of value estimation, and examine how privileged information affects policy learning. The main contribution of the paper is an extension of established recurrent actor-critic theory to this asymmetric setting. No experiments are reported, this makes the validation entirely theoretical.

Critical strengths

The theoretical question is quite interesting as understanding when training-only information helps a recurrent critic could guide the design of methods for partially observable environments.

Critical weaknesses
  • I believe the Bellman equation in Proposition E.3 is invalid which undermines the proof of Theorem 5.1. The proposition assumes that the next critic input preserves the information needed to predict future returns. However, as per the definition of the history (line 105), ht+1h_{t+1} does not retain the previous privileged information iti_t. Eq. (13) therefore does not follow.

    This can be shown in a two-state informed-POMDP counterexample with one action, a constant observation, uniformly distributed S∈−1,+1S\in\\{-1,+1\\}, reward rt=Sr_t=S, and discount factor γ=1/2\gamma=1/2. Each privileged information It∈−1,+1I_t\in\\{-1,+1\\} is independently sampled conditional on SS, with probability 3/43/4. Then the LHS and RHS of Proposition E.3 are:

    \qquad \mathbb E[r_t+\gamma Q^\pi(h_{t+1},i_{t+1})\mid i_t] =\frac58 i_t.$$
  • A separate weakness is the lack of experiments. The paper proposes an algorithm and discusses a trade-off between the benefits of privileged information and the additional learning complexity, but provides no empirical evidence showing when either effect matters. At minimum, a controlled comparison with a symmetric recurrent critic would help establish the significance of the analysis for value estimation and policy learning.

Questions for authors

Regarding the critical weakness above, could the authors provide any additional assumption ensuring that Proposition E.3 remains valid when the previous privileged signal is discarded?

3 comments

We thank the reviewer for their valuable feedback and for recognizing the relevance of the main theoretical question addressed in our work. In response to the review, we have revised the manuscript to address the concerns raised. For ease of reference, all significant changes in the revised manuscript are highlighted in blue. Below, we respond to each comment in detail.

W1/Q1: Bellman equation in Proposition E.3. We thank the reviewer for this important finding, and we confirm that their counterexample is correct for the original statement. Proposition E.3, as originally stated, implicitly treated (ht,it)(h_t, i_t) as evolving according to a one-step Markov structure, which does not hold for arbitrary privileged signals it∼I(⋅∣st)i_t \sim I(\cdot \vert s_t) sampled independently at each time step tt. We have fixed this by introducing an explicit condition (Assumption 3 in Section 2.3 of the revised manuscript) under which the recursion is valid.

Proposition E.3 is now stated conditionally on Assumption 3, which requires that the previous privileged signal iti_t carries no information about future returns beyond what is already captured by (ht+1,it+1,at+1)(h_{t+1}, i_{t+1}, a_{t+1}). That is, almost surely, Eπ[∑j≥1γj−1rt+j∣ht+1,it+1,at+1,ht,it,at]=Eπ[∑j≥1γj−1rt+j∣ht+1,it+1,at+1].\mathbb{E}^\pi\left[\sum_{j \geq 1} \gamma^{j-1} r_{t+j} \vert h_{t+1}, i_{t+1}, a_{t+1}, h_{t}, i_{t}, a_{t}\right] = \mathbb{E}^\pi\left[\sum_{j \geq 1} \gamma^{j-1} r_{t+j} \vert h_{t+1}, i_{t+1}, a_{t+1}\right].

Under this assumption, Eq. 13 follows exactly as in the revised proof of Proposition E.3, and all downstream results (the architecture, Lemma 5.1, Theorem 5.1, Propositions 5.1-5.2) hold as originally stated.

The reviewer's counterexample is consistent with the revised statement. In their example, the previous signal iti_t does carry information about future returns beyond (ht+1,it+1,at+1)(h_{t+1}, i_{t+1}, a_{t+1}), since the latent state is persistent and the signals are noisy snapshots. Hence, Assumption 3 is violated in this example, and Proposition E.3 makes no claim in this regime. However, the assumption is satisfied by the privileged signals used in the vast majority of asymmetric actor-critic applications:

  • Full state: it=sti_t = s_t, or any signal containing it (Pinto et al., 2017; Baisero & Amato, 2022; Degrave et al., 2022; Kaufmann et al., 2023; Vasco et al., 2024). Future returns are determined by (st+1,at+1)(s_{t+1}, a_{t+1}) and subsequent transitions; the past state sts_t adds nothing given st+1s_{t+1}.
  • Static latent quantities: it+1=iti_{t+1} = i_t. Examples: goal location or target waypoint in navigation tasks; fixed physical parameters of the environment such as object mass, terrain friction, or motor strength in locomotion; the task identity or a task embedding in multi-task/meta-RL; the adversary's fixed policy parameters in competitive games. In all of these, the signal is constant across time, so iti_t is trivially contained in it+1i_{t+1}.
  • Self-refining signals: it+1i_{t+1} is at least as informative as iti_t, e.g., it+1=(it,ot+1+)i_{t+1} = (i_t, o_{t+1}^+). Such a self-refining sequence is equivalent to making no assumption at all and feeding the accumulated signal i0:ti_{0:t} to the critic. So, in principle, the theory covers arbitrary privileged signals, including the reviewer's example, at the price of a growing privileged input. Growing inputs are, however, of limited practical value for the proposed critic architecture. An input iti_t that grows with tt would require a correspondingly growing parameter vector bb, which a fixed-size network cannot accommodate. In practice, one therefore either uses signals from the previous two cases, or compresses accumulating signals into a fixed-dimensional sufficient statistic (next case). We view the handling of growing privileged inputs, e.g., via a recurrent privileged branch with nonlinear integration, as future work.
  • Return-sufficient latents with next-state-sufficient abstraction: it=ϕ(st)i_t = \phi(s_t) where ϕ\phi is a predictive sufficient statistic, so that ϕ(st)\phi(s_t) is redundant given ϕ(st+1)\phi(s_{t+1}). Examples: in cart-pole swing-up or pendulum balancing with pixel observations, the pole angle and angular velocity form a dynamically closed, reward-relevant abstraction from which the future trajectory of these two quantities, and thus the return, is determined. This case also covers accumulated signals compressed into a fixed-size sufficient statistic, e.g., a posterior over a persistent latent variable.

We note that when it=oti_t = o_t (the standard POMDP setting), Assumption 3 reduces to the standard belief model, so the symmetric theory of Cayci & Eryilmaz (2025) is recovered exactly. Finally, we agree with the reviewer that signals providing fresh, non-cumulative noisy observations of a persistent latent variable, as in their example, are not covered by the assumption's practical scope.

W2: Lack of experiments. We understand that empirical validation would strengthen the paper by illustrating the theoretical insights, and we appreciate the concrete suggestion of a controlled comparison with a symmetric recurrent critic. In this work, however, we focused on a clean theoretical treatment of asymmetric Rec-NAC with state-conditioned privileged information and did not include empirical experiments. We agree that a small-scale controlled study is well aligned with the type of validation our theory invites and is a natural next step. To address this concern, we propose to include a small, controlled T-maze POMDP in the revised manuscript.

Environment. In this environment, a cue is presented only once and must be remembered throughout the episode. We describe the environment and our preliminary results below. These results provide empirical support for the theoretical claims of our paper.

An episode lasts T=L+1T=L+1 steps. At t=0t=0 the following static variables are drawn:

  • the cue c∈−1,+1c\in\\{-1,+1\\}, indicating the target arm;
  • an indicator variable f∈−1,+1f\in\\{-1,+1\\} that encodes whether the cue signal is flipped, with P(f=−1)=p\mathbb{P}(f=-1)=p;
  • a nuisance variable z∈−1,+1z\in\\{-1,+1\\}.

The full state at time tt is st=(t,c,f,z)s_t=(t,c,f,z). The observation is ot=(cf⋅1[t=0],tL,1[t=L])o_t=(cf\cdot\mathbb{1}[t=0], \frac{t}{L},\mathbb{1}[t=L]), so that the (possibly flipped) cue is visible only at t=0t=0 and must be remembered for the subsequent LL steps. The remaining observation components indicate the agent's position and whether it has reached the junction. The actions are forward,left,right\\{\text{forward},\text{left},\text{right}\\}: only forward is valid for t<Lt<L, and only left/right at the junction t=Lt=L.

The reward is rt=0r_t=0 for t<Lt<L and rL=±1+κzr_L=\pm1+\kappa z, with +1+1 if the chosen arm matches cc and −1-1 otherwise. Choosing the left arm corresponds to c=−1c=-1. The nuisance term κz\kappa z with κ∈0,1\kappa \in\\{0,1\\} is independent of the action. The best achievable success probability is 1−p1-p.

The informed variant provides noise-free privileged signals satisfying Assumption 3. This allows us to derive QπQ^\pi, the information-aliasing error, and the advantage function in closed form. We consider the following privileged signals:

  • state: it=st=(t,c,f,z)i_t=s_t=(t,c,f,z).
  • static: it=ci_t=c, revealing the (not-flipped) goal location.
  • abstraction: it=ϕ(st)=(c,γL−tz)i_t=\phi(s_t)=(c, \gamma^{L-t}z), which discards tt and ff.

Note that the privileged signal may reveal only zz, only cc, or both. When only zz is revealed, Qπ(h,i,a)Q^\pi(h,i,a) is additive in ii (Assumption 4). When cc is revealed, the signal also identifies the correct arm, so QπQ^\pi contains interaction between aa and ii.

Results.

Our experiments examine the following aspects of the theoretical analysis: (i) the information-aliasing error floor of the symmetric critic, (ii) variance reduction in the critic's regression target, (iii) advantage-level information mismatch under additive realizability, and (iv) the additional complexity cost for the informed critic.

(i) Information-aliasing error floor.

Our preliminary policy evaluation results with Rec-TD (L=8L=8, p=0.1p=0.1, mF=128m_F=128, Ktd=1000K_{\text{td}}=1000; mean ±\pm std over 10 independent runs) show that the informed critic eliminates the information-aliasing error in the additive regime. In contrast, it provides no benefit when the privileged signal is constant and cannot resolve the aliasing. These observations are consistent with our theoretical results.

iti_tAliasing error along the trajectorySymmetric MSEInformed MSE
(t,z)(t, z)4.906 ±\pm 0.0005.045 ±\pm 0.0184.701 ±\pm 0.366
γL−tz\gamma^{L-t} z4.906 ±\pm 0.0005.045 ±\pm 0.0183.948 ±\pm 0.850
(c,γL−tz)(c, \gamma^{L-t} z)6.170 ±\pm 0.0666.308 ±\pm 0.0755.055 ±\pm 0.553
cc1.251 ±\pm 0.0391.389 ±\pm 0.0381.404 ±\pm 0.046

(ii) Variance reduction in the critic's regression target.

The following table reports the variance decomposition of the TD-target variance at the junction t=Lt=L (L=8L=8; mean ±\pm std over 10 independent seeds). it=∅i_t=\emptyset corresponds to the symmetric setting.

iti_tVar(yL∣hL,aL)\mathrm{Var}(y_L \vert h_L,a_L)E[Var(yL∣hL,aL,iL)]\mathbb{E}[\mathrm{Var}(y_L \vert h_L,a_L,i_L)]Var(E[yL∣hL,aL,iL]∣hL,aL)\mathrm{Var}(\mathbb{E}[y_L \vert h_L,a_L,i_L] \vert h_L,a_L)
∅\emptyset1.640 ±\pm 0.0021.640 ±\pm 0.0020.000 ±\pm 0.000
(t,z)(t, z)1.640 ±\pm 0.0020.640 ±\pm 0.0001.000 ±\pm 0.002
γL−tz\gamma^{L-t} z1.640 ±\pm 0.0020.640 ±\pm 0.0001.000 ±\pm 0.002
(t,c,f,z)(t, c, f, z)1.640 ±\pm 0.0020.000 ±\pm 0.0001.640 ±\pm 0.002
(c,γL−tz)(c,\gamma^{L-t} z)1.640 ±\pm 0.0020.000 ±\pm 0.0001.640 ±\pm 0.002
cc0.640 ±\pm 0.0000.000 ±\pm 0.0000.640 ±\pm 0.000

(iii) Advantage-level information mismatch.

The following table reports the information-mismatch errors for L=8L=8 and p=0.2p=0.2, using history-only actors with mG=64m_G=64 (mean ±\pm std over 10 independent runs).

iti_tεinfo\varepsilon_{\text{info}}εinfoA\varepsilon^{A}_{\text{info}}∑tγtE[ξˉt2]\sum_t \gamma^t \mathbb{E}[\bar\xi_t^2]
(t,z)(t, z)4.906 ±\pm 0.0000.000 ±\pm 0.0004.906 ±\pm 0.000
γL−tz\gamma^{L-t} z4.906 ±\pm 0.0000.000 ±\pm 0.0004.906 ±\pm 0.000
(t,c,f,z)(t,c, f, z)5.337 ±\pm 0.0110.422 ±\pm 0.0024.915 ±\pm 0.009
(c,γL−tz)(c, \gamma^{L-t} z)5.337 ±\pm 0.0110.422 ±\pm 0.0024.915 ±\pm 0.009
cc0.432 ±\pm 0.0100.422 ±\pm 0.0020.010 ±\pm 0.010

These results illustrate the distinction between the QQ-based and advantage-level information-mismatch errors. When iti_t enters additively through the nuisance variable zz, the advantage-level error vanishes, even though the QQ-based error can remain positive. When the signal also reveals the cue cc, the advantage-level error is nonzero because the privileged information affects the action-dependent part of the value function.

(iv) Complexity costs.

To examine the effect of privileged-signal dimensionality, we vary its dimension did_i by padding it with redundant copies. The following table reports the MSE gap between the informed and symmetric critics for two settings with L=8L=8 (Ktd=1000K_{\text{td}}=1000; mean ±\pm std over 10 independent runs). The privileged signal is normalized as described in the paper.

  • Setting 1: static signal in the general regime (p=0p=0, κ=0\kappa=0). There is no information aliasing, so the informed critic can only incur an additional complexity cost. The symmetric-critic MSE is 0.217±0.0060.217 \pm 0.006.
  • Setting 2: abstraction signal in the additive regime (p=0.2p=0.2, κ=1\kappa=1). Information aliasing is present; the symmetric-critic MSE is 4.973±0.0034.973 \pm 0.003 with an aliasing floor of 4.9064.906.
d_iSetting 1Setting 2
1+0.003 ±\pm 0.012-1.671 ±\pm 0.593
16+0.005 ±\pm 0.010-3.301 ±\pm 0.357
64+0.008 ±\pm 0.006-3.047 ±\pm 0.196
256+0.019 ±\pm 0.009-1.901 ±\pm 0.248

Without information aliasing, the informed critic performs worse than the symmetric critic. This performance gap increases with did_i and decreases with training length. For di=256d_i=256, it decreases from +0.090±0.039+0.090\pm0.039 at Ktd=100K_{\text{td}}=100, to +0.026±0.008+0.026\pm0.008 at Ktd=300K_{\text{td}}=300, and +0.019±0.009+0.019\pm0.009 at Ktd=1000K_{\text{td}}=1000. This is consistent with a complexity term that scales with the privileged-signal dimension and decays with KtdK_{\text{td}}.

When information aliasing is present, the informed critic outperforms the symmetric critic at every tested dimension, with MSEs ranging from 1.6721.672 to 3.3023.302 compared with 4.9734.973 for the symmetric critic. The performance gain generally increases with training length.

We also conducted experiments without normalizing the privileged signal, in which its norm grows as sup⁡∣∣i∣∣2=λ⋅di\sup||i||_2=\lambda\cdot\sqrt{d_i}. At a large scale of λ=10\lambda=10, training diverges for di≥16d_i\ge16. The performance gaps are +72±92+72\pm92 for di=16d_i=16, +129±126+129\pm126 for di=64d_i=64, and +290±304+290\pm304 for di=256d_i=256. This is consistent with the dependence on sup⁡∣∣i∣∣2\sup||i||_2 in the step-size condition of our analysis; normalization removes this dependence on signal magnitude.

We hope these answers fully settle all open questions. Please do not hesitate to ask us to elaborate further if anything remains unclear.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.