- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper extends Recurrent Natural Actor-Critic to an asymmetric setting in which the critic receives state-dependent privileged information while the actor observes only interaction histories. It presents finite-time and finite-width bounds for recurrent TD learning, separates information and representation effects, and analyzes how informed value estimation enters the actor’s compatible function approximation error.
Critical strengths
- Useful separation of information, representation, and complexity effects.
The paper distinguishes genuinely additional information from a more accessible representation of information already contained in the history. This provides a useful framework for understanding why privileged inputs need not yield uniform improvements under finite computational and statistical budgets.
- Explicit connection between critic estimation and actor-side approximation.
The analysis separately tracks optimization, finite-width, and truncation contributions, and connects critic estimation error to compatible regression error. This organization makes the assumptions and potential bottlenecks more explicit than a purely asymptotic convergence statement.
Critical weaknesses
- The Bellman relation is not justified for arbitrary state-dependent privileged signals.
Section 2.3 and Proposition E.3 assert
\mid h_t,i_t,a_t \right]$$ However, the next critic input retains only the current privileged signal $i_{t+1}$, not the previous signal $i_t$. The stated information-generation model does not guarantee that $i_t$ becomes irrelevant to future returns after conditioning on $(h_{t+1},i_{t+1},a_{t+1})$. In particular, writing $G_{t+1}$ for the remaining discounted return, the assumptions do not establish $$\mathbb E[G_{t+1}\mid h_{t+1},i_{t+1},a_{t+1},i_t]=\mathbb E[G_{t+1}\mid h_{t+1},i_{t+1},a_{t+1}].$$ For a persistent hidden state observed through independently noisy privileged measurements, the discarded measurement can still contain predictive information. Consequently, the tower property alone does not justify the proposed recursion. This affects the central convergence claim: Eq. (13) in the proof of Theorem 5.1 explicitly uses this Bellman relation. An additional sufficiency assumption, or a formulation retaining the privileged history, appears necessary before the TD fixed point and convergence analysis can be established in the claimed generality. 2. **The proposed architecture removes privileged dependence from the advantage, weakening the interpretation of actor-side information mismatch.** The critic in Section 3.2 is additive: $$\hat Q_t(h,i,a)=g_t(h,a)+\beta^\top i.$$ Applying the advantage definition in Section 4.2 therefore gives $$\hat A_t(h,i,a)=g_t(h,a)-\sum_{a'}\pi(a'\mid h)g_t(h,a'),$$ because the privileged term cancels exactly. Thus, for fixed critic parameters, the actual compatible-regression target does not depend on i. Privileged information can still affect the learned history branch indirectly through TD training; this cancellation does not make the algorithm equivalent to symmetric training. Nevertheless, it creates a substantial disconnect between the implemented architecture and the explanation that the actor must fit privileged-dependent advantage variation. Under exact realizability in the corresponding additive class, the same cancellation holds for the true advantage. Moreover, the proposed $\varepsilon_{\mathrm{info}}$ measures conditional variation in $Q^\pi$, rather than in the advantage, and can therefore count action-independent baseline variation that is irrelevant to compatible regression. The analysis should distinguish this potentially loose upper-bound contribution from genuine actor-side approximation difficulty.Questions for authors
Why are no numerical experiments included? Could small, controlled POMDP experiments test the predicted information–complexity trade-off and whether improved critic accuracy translates into better policy performance?
4 comments
W2: Privileged dependence cancels in the advantage estimate. We thank the reviewer for this observation, and indeed the privileged block cancels in the estimate of the informed advantage function: is -independent for every fixed critic parameter. Privileged information thus reaches the compatible regression only indirectly, through the TD training of the recurrent representation ; we agree that this does not make the algorithm equivalent to symmetric training.
We view this cancellation as a positive structural property of the proposed architecture, which yields the best of both worlds: on the critic side, privileged information reduces the variance of the TD target (Proposition E.1) and removes the information-aliasing error (Lemma 5.1); on the actor side, the same privileged variation does not enter the compatible-regression target as additional noise. In particular, the second moment of the informed advantage target is not affected by additional variation due to the privileged signal, and the actor-side difficulty reduces to the actor’s expressive capacity for the history-measurable target, evaluated at the TD-learned history representation of the informed critic, together with and . Intuitively, the critic exploits what the actor cannot see, while the actor's learning problem is not burdened by variation it could never fit.
We have made the following changes to the manuscript to account for this finding (marked in blue in the revision): Remark 4.1 (proof in Lemma E.2, Appendix E.6) states the -independence of the advantage estimate; Lemmas E.3-E.4 prove the baseline invariance and the orthogonality of the privileged value deviation; Corollary E.4 shows that the advantage-level floor vanishes under realizability; Remark E.1 identifies the -based error with the information-aliasing error; and Section 5.3 states explicitly that should be read as a conservative surrogate, distinct from genuine actor-side approximation difficulty.
Regarding the -based vs. advantage-based target (the former mirrors the definition for the symmetric setting in Cayci and Eryilmaz (2025)): the reviewer is correct that being defined w.r.t. can include action-independent baseline variation irrelevant to compatible regression. This looseness is not harmful. By the baseline invariance of compatible function regression, the -based and advantage-based objectives differ by a -independent constant, without biasing the update direction. Formally, , where the advantage-level information-mismatch error isolates the genuine action-dependent privileged variation and the second term cancels in the advantage. Under realizability in the additive class (Assumption 4 in the revised manuscript), the true informed advantage is -independent almost surely, so , and Remark E.1 gives gives the exact identity ; the conditional variance removed on the critic side is counted on the actor side instead, without changing the regression minimizer. The advantage-level error term, becomes non-zero only in the non-realizable regime or for critics with action-dependent privileged coupling; we leave the analysis of latter to future work. For a more detailed discussion, we refer to Section 5.3 in the revised manuscript.
Q1: Lack of numerical experiments. This point overlaps with the concern raised in W2 by Reviewer XD5L; we refer to our response there. In brief: we focused this work on a clean theoretical treatment, and we agree that small, controlled POMDP experiments testing the described information–complexity trade-off are well aligned with the validation our theory invites. We propose to add experiments on a small T-maze POMDP in which a cue is presented only once and must be remembered.
We hope these answers fully settle all open questions. Please do not hesitate to ask us to elaborate further if anything remains unclear.
Thanks for the detailed responses and the additional experiments, which address most of my concerns.
However, I believe the manuscript still needs some clarification and revision:
Scope of the theoretical claims: The new assumption 3 substantially restricts the class of admissible privileged signals. However, the manuscript still describes the setting such as arbitrary state-dependent privileged signals.
Policy-level validation: The new experiments mainly validate critic estimation and the proposed error decompositions. It would be helpful to clarify that these results do not yet establish improvements in end-to-end policy learning or final policy performance.
We thank the reviewer once again for their valuable feedback, and for listing their remaining concerns. Below, we address these concerns.
Scope of the theoretical claims.
We agree that Assumption 3 restricts the class of admissible state-dependent privileged signals and that describing our setting as allowing arbitrary state-dependent privileged signals is therefore too broad. This imprecise wording carried over from the original submission. We have now revised the corresponding statements throughout the manuscript to clarify that our analysis concerns state-dependent privileged signals satisfying Assumption 3. In particular, we have made this explicit in the Introduction (Section 1), the analysis (Section 5), and the Conclusion (Section 6).
Policy-level validation.
We agree that the experiments presented in our response to Reviewer XD5L primarily validate the critic-side theoretical results (variance reduction, elimination of information-aliasing error, and additional complexity costs) and the actor-side approximation error decomposition; since these are the main results of our theoretical analysis. As the reviewer correctly notes, they do not establish improvements in end-to-end policy learning or final policy performance. We also do not make such claims for asymmetric Rec-NAC in the manuscript. In fact, we explicitly state that improvements in value estimation do not automatically translate into uniformly tighter policy-learning guarantees when the critic has access to information unavailable to the actor (cf. Section 6).
To complement our empirical results, we have extended our experiments to the full Rec-NAC learning cycle, including policy optimization. The following table reports preliminary results on a corridor of length with and (, , , ), averaged over ten independent runs (mean standard deviation). AUC denotes the normalized area under the mean empirical episodic return throughout training. Final return, success rate, and critic's MSE are averaged over the last five policy iterations. corresponds to the symmetric setting.
| AUC (mean return) | final return | success rate | critic's MSE | |
|---|---|---|---|---|
| 0.381 0.060 | 0.447 0.077 | 0.722 0.037 | 5.171 0.042 | |
| 0.389 0.071 | 0.458 0.084 | 0.731 0.041 | 4.614 0.161 | |
| 0.389 0.061 | 0.469 0.078 | 0.731 0.036 | 2.641 0.389 |
The informed critic achieves comparable policy performance to the symmetric critic. Although mean performance is slightly higher for the asymmetric variants with access to the full-state and abstraction signals, the differences are not substantial. These findings are consistent with our theoretical analysis. For the proposed critic architecture, the privileged branch cancels in the advantage estimate, so privileged information does not directly alter the actor's regression target. Instead, it can affect policy learning indirectly through improvements in the critic's history representation.
To conclude, our theoretical analysis characterizes how critic and advantage-estimation errors enter the policy update, but does not establish that improved value estimation necessarily translates into better policy performance, particularly under limited training budgets. We will make this explicit in the camera-ready revision.
We hope that these clarifications and additional results address your remaining concerns.
We thank the reviewer for their valuable feedback and for highlighting the key contributions of our work. In response to the review, we have revised the manuscript to address the concerns raised. For ease of reference, all significant changes in the revised manuscript are highlighted in blue. Below, we address each concern individually.
W1: Bellman relation in Proposition E.3. We thank the reviewer for catching this issue and confirm that the observation is correct for the original statement: the tower property alone does not justify the recursion when the next critic input retains only it+1 and not it. The requirement E[Gt+1∣ht+1,it+1,at+1,it]=E[Gt+1∣ht+1,it+1,at+1] is precisely the assumption that was needed, but not made explicit, in the original submission. We have fixed this issue in the revised manuscript by making this requirement explicit (Assumption 3 in Section 2.3): past privileged signals must be redundant for future returns once (ht+1,it+1,at+1) is observed. Proposition E.3, and with it Eq. (13) in the proof of Theorem 5.1 and all downstream convergence results, are now stated and proven under this assumption; the results hold as originally claimed within this scope. Importantly, Assumption 3 is satisfied by the signal classes used in the vast majority of asymmetric actor-critic applications (full state, static latents, self-refining signals, and return-sufficient abstractions such as ϕ(st)). For a detailed discussion and examples of common privileged signals satisfying Assumption 3, we refer to our response to W1/Q1 of Reviewer XD5L.
Regarding the alternative fix suggested by the reviewer: retaining the privileged history as critic input indeed makes the sequence self-refining, so that the recursion holds without any assumption. However, the resulting input i0:t grows with time t and cannot be accommodated by the fixed-size linear readout of the proposed architecture; in practice, the sequence of past privileged signals must be compressed into a fixed-sized representation before being fed to the critic (cf. the discussion after Assumption 3 in the revised manuscript). Moreover, the particular choice of critic architecture made in this work allows us to eliminate the actor-side advantage-level information-mismatch error, as now made clear in Lemma E.2, Lemma E.4, and Corollary E.4 in Appendix E.6. We discuss this point in detail in our response to W2. We view critic architectures that can natively handle growing privileged inputs, e.g., via a recurrent privileged branch, as future work.