acceptodds
Under review as a conference paper at ICLR 2027

Online Extraction of Privileged Signals for Asymmetric Actor-Critic

Abstract

Asymmetric actor-critic methods let the critic, but not the actor, use privileged information during training; which information the critic sees is decided offline, in Informed Asymmetric Actor-Critic (IAAC) by how well it predicts returns on episodes of a fixed policy. We ask what that information does to the actor's update as the policy changes, and whether choosing it online helps. We split what a signal tells the critic into an offset shared by all actions and a contrast between them, and derive the update cost of each. The offset removes noise, but its interaction with the contrast can reverse the benefit: with two actions, an independent signal that shifts the return of the action the policy commits to makes an exact value baseline's update noisier once that action's probability exceeds two thirds, although the signal predicts returns better. A signal's value is thus not stationary, and we propose Online Informed Actor-Critic (OIAC), which re-selects the critic's input during training by a paired confidence bound on the update noise and keeps cross-fitted Monte Carlo updates conditionally unbiased. For bootstrapped advantages, the future offsets add back what the baseline removes, so a one-step actor receives mostly the contrast. In finite informed POMDPs solved exactly, the signal that IAAC's criterion selects lowers the second moment of a Monte Carlo update by up to 64% but raises that of a one-step update by up to 210%. In tabular learning, only OIAC among four selection criteria dropped a signal once it turned harmful; in that example it learned 13% faster than the offline choice under normalized steps but stayed slower than a critic without privileged information, and it was not faster under plain SGD, whereas the exact variance-optimal baseline made the same signal useful.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Asymmetric Actor-Critic

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_oxNsOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies which privileged signals should be provided to the critic in asymmetric actor-critic methods and whether these inputs should adapt during training. The authors show that improving the critic’s return prediction does not necessarily benefit actor learning: a previously useful signal can increase update noise as the policy changes. The analysis decomposes a signal’s effect into a shared value offset and a change in relative action values, and examines their different effects on Monte Carlo and one-step TD updates. Based on this analysis, OIAC selects critic inputs online using an update-noise statistic. Experiments in small synthetic environments show that it can remove signals that become harmful during learning. However, noise reduction does not consistently accelerate learning, and benefits depend on the optimizer and baseline design. OIAC’s learning experiments are limited to tabular models with Monte Carlo updates.

Critical strengths
  1. The problem is important and the mechanism analysis is clear. The paper distinguishes accurate return prediction by the critic from useful actor updates. Its offset–contrast decomposition explains why the effect of privileged information changes with the policy and advantage estimator, providing concrete insight into when privileged information helps or hurts asymmetric actor-critic learning.

  2. Controlled experiments support the analysis, and the findings are carefully qualified. The authors use exact calculations and controlled experiments to examine the mechanism, while explicitly reporting that noise reduction need not accelerate learning, benefits depend on the optimizer, and harmful signals may be detected late. This helps distinguish evidence for the mechanism from evidence of practical optimization gains.

Critical weaknesses
  1. OIAC’s practical advantage over simpler alternatives remains insufficiently established. In the main one-decision experiment, OIAC requires approximately 13% fewer episodes than IAAC under normalized updates, but approximately 8% more than the history-only critic. It shows no detectable speed-up under plain SGD, and the small Adam gain does not replicate on the confirmation seeds. Pooled estimates of the variance-optimal baseline also achieve substantial gains. These results leave the independent value of OIAC’s additional selection complexity unclear relative to these alternatives.

  2. The algorithm does not yet address an important setting motivating the paper. Although the analysis emphasizes that the same signal can have opposite effects on Monte Carlo and TD updates, OIAC’s selection statistic and learning experiments remain restricted to Monte Carlo. Its applicability to bootstrapped actor-critic methods is therefore unestablished, and the experiments are confined to small synthetic environments and tabular models. While the insights built on established baseline-variance theory and advantage decompositions are informative, I find the incremental theoretical contribution insufficient to offset the current limitations in algorithmic scope and demonstrated benefit.

Questions for authors
  1. When does online signal selection offer a practical advantage?
    With comparable data access, estimation protocols, tuning effort, and computational budgets, under what conditions, if any, does OIAC outperform a history-only critic or a pooled estimate of the variance-optimal baseline b⋆b^\star? Which existing results support such an advantage?

  2. How does the selection criterion extend to shared-parameter actors and bootstrapped updates?
    For shared-parameter actors, the current statistic omits cross-step covariance terms; with TD/GAE, learned critics can also introduce bootstrap-induced bias. How would these factors affect the appropriate selection criterion and the ranking of candidate signals? What conditions or evidence would support reliable rankings beyond the current tabular Monte Carlo setting?

  3. What is the strongest practical evidence for adopting OIAC?
    Given that the Adam speed-up did not replicate on the confirmation seeds, which demonstrated benefits provide the main practical justification for adopting OIAC, beyond the observed reduction in update noise?

Reviewer_iG4mOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The submission addresses asymmetric reinforcement learning for partially observable control, and propose to select different forms of information based on prediction and performance criteria.

Critical strengths

The subject matter would be interesting, were it comprehensible.

Critical weaknesses

Readability:

Despite being extremely familiar with the subject matter, I am unable to understand most of the text in the submission.

Some non-comprehensive examples of sentences that are extremely hard to understand:

  • lines 17-20: "with two actions, an independent signal that shifts the return of the action the policy commits to makes an exact value baseline’s update noisier once that action’s probability exceeds two thirds, although the signal predicts returns better."
  • lines: 28-31: "in that example it learned 13% faster than the offline choice under normalized steps but stayed slower than a critic without privileged information, and it was not faster under plain SGD, whereas the exact variance-optimal baseline made the same signal useful."
  • lines 92-95: "It assesses candidate signals on episodes of a fixed policy, independently of the actor’s training: a residual-independence test before training, or the gain in held-out return prediction afterwards, retains feature subsets"
  • lines 104-107: "The variance-optimal baseline weights each action by its squared score (...), with two actions by swapping their probabilities (...), also with information the policy does not see (...), and baselines also change how fast a softmax policy commits (Chung et al., 2021; Mei et al., 2022)"
  • lines 108-109: "Our sign rules concerns the value baseline that asymmetric methods actually fit, and shows when a more predictive signal increases its update noise."
  • line 112: "we add when the offset cancels and what a learned informed critic’s error does to the actor"
  • Figure 1: The whole description is hard to understand and lacks important details. What environment is this used on? What are the uu and ww signals?

To say the least, I am unable to provide further feedback due to the presentation issues.

The submission requires a thorough invasive rewrite before being considered for publication. I invite the authors to perform such rewrite carefully, focusing on human comprehension.

Questions for authors

N/A

Reviewer_DdzhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This is primarily a theoretical analysis of asymmetric actor-critic, where the critic (not the actor) sees privileged information during training. Prior work selects the critics signal offline by how well it predicts returns. The authors point is that the actor does not use the critic prediction, it uses the critic's advantage, so return-prediction quality is the wrong selection objective. They decompose what a signal adds to the critic into an offset and a contrast, and derive the per-sample update cost for both the advantage and baseline forms.

Critical strengths
  1. A genuine conceptual correction, rigorously established
  2. Exemplary methodological honesty.
Critical weaknesses
  1. The proposed method under delivers, by the authors own evidence. OIAC only avoids a harmful signal; it never makes a signal useful, is dominated by the (exact) variance-optimal baseline, and in the setting where IAAC's signal is actually beneficial it mostly just reproduces IAAC's choice. Its net practical advantage over simply using a history-only critic or fixing the baseline weights is unclear. For a submission proposing a method (OIAC), this is a real limitation.
  2. The evidence is confined to tabular tasks, and the method may not transfer to the regime that matters.
Questions for authors
  1. Given Prop. 3.4, what is the intended use of OIAC with bootstrapped/learned critics? Do you have even preliminary evidence that a bootstrapped-form selection statistic (which you name as future work) recovers the benefit, or should practitioners instead prefer an estimated variance-optimal baseline?
  2. Does the sign rule extend beyond two actions and beyond the single-action additive-shift signal model, at least directionally? A statement of how it generalises would materially affect how impactful the analysis is.
  3. OIAC's statistic is dominated by rare-action episodes, causing late detection and re-admission. Can you characterise when this late detection is too late to matter on a non-tabular task?
Reviewer_EKzgOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This is a very nice paper about Informed Asymmetric Actor-Critic (IAAC) and in particular a key aspect related to those algorithms: how to select the relevant information. The paper comes with an informative theoretical analysis and compétent simulation results.

Critical strengths

Very interesting topic.

Valuable theoretical analysis.

Competent simulation results.

Critical weaknesses

I did not see critical weaknesses.

Questions for authors

An advice more than a question:

Try to come up with a very small benchmark on which you could better illustrate your findings. I think it would help to build a community around your work and others in the field of IACC.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Comment to Reviewers

Dear Reviewers,

Thank you all for the time and thoughtful feedback you have devoted to our paper. We have done our best to address each of your comments in our responses and in the revised manuscript. In particular, following your suggestions, the latest revision frames the paper as an analysis with OIAC as a minimal test and clearly states the scope and limitations of the evidence in the abstract.

We understand how busy your schedules are, but as the discussion period is closing soon, we would greatly appreciate it if you could take a moment to review our responses and the latest revision. We remain happy to answer any remaining questions.

Best regards, The Authors

AuthorsOpenReview ↗

General Response: Summary of Revisions

For the convenience of the reviewers and the AC, we summarize the revisions made during the discussion period below. The revised paper frames the work as an analysis of when a privileged signal helps or hurts the actor's update, with Online Informed Actor-Critic (OIAC) as a minimal test of the analysis.

Changes in the revised paper

ChangeReviewersWhere
Improved readability only, without changing the content: rewrote hard-to-follow passages, reworded the rest of the main text, and added an environment diagramiG4mAbstract, Secs. 1–2, 3.1–3.2 (rewritten); Secs. 3.3–6 (reworded); Fig. 3
Framed the paper as an analysis, with OIAC as a minimal test, and stated the limits of the evidence in the abstractDdzh, oxNsAbstract, Sec. 1, Secs. 4–6
General sign rule for any action count and signal (exact critics), with the softmax threshold for KK actionsDdzh, oxNsCor. 3.4, Eq. (11), App. A.7
Clarified when offsets cancel under bootstrapping and how critic error biases the actorDdzh, oxNsProp. 3.5, Eq. (14)
Episode-level statistic for shared parameters (untested)oxNsSec. 4
Comparison of batch vs. pooled (persistent) criticsDdzh, oxNsSec. 5.4, App. E.3, Table 11
Study on 11 classic POMDPs (60 seeds)EKzg, Ddzh, oxNsSec. 5.4, App. E.4, Fig. 4, Tables 12–13
Practical implications: a history-only critic or a pooled b⋆b^\star as a strong, simpler alternativeDdzh, oxNsSec. 6
Environments and exact evaluator to be released as a small benchmarkEKzgReproducibility statement

Points we could not complete in the discussion period

PointReviewersStatus
Neural networks and bootstrapped actor-critic with learned criticsDdzh, oxNsNot tested in this paper. Sec. 6 (Limitations and future work) states this as a limitation and leaves it to future work.

Final version

If needed and given the opportunity, we will further improve the paper's presentation and readability in the revised version—including notation, figure and table layout, and sentence-level clarity—without changing its content, claims, or results.

We thank the reviewers and the AC again for their time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.