Online Extraction of Privileged Signals for Asymmetric Actor-Critic

Asymmetric actor-critic methods let the critic, but not the actor, use privileged information during training; which information the critic sees is decided offline, in Informed Asymmetric Actor-Critic (IAAC) by how well it predicts returns on episodes of a fixed policy. We ask what that information does to the actor's update as the policy changes, and whether choosing it online helps. We split what a signal tells the critic into an offset shared by all actions and a contrast between them, and derive the update cost of each. The offset removes noise, but its interaction with the contrast can reverse the benefit: with two actions, an independent signal that shifts the return of the action the policy commits to makes an exact value baseline's update noisier once that action's probability exceeds two thirds, although the signal predicts returns better. A signal's value is thus not stationary, and we propose Online Informed Actor-Critic (OIAC), which re-selects the critic's input during training by a paired confidence bound on the update noise and keeps cross-fitted Monte Carlo updates conditionally unbiased. For bootstrapped advantages, the future offsets add back what the baseline removes, so a one-step actor receives mostly the contrast. In finite informed POMDPs solved exactly, the signal that IAAC's criterion selects lowers the second moment of a Monte Carlo update by up to 64% but raises that of a one-step update by up to 210%. In tabular learning, only OIAC among four selection criteria dropped a signal once it turned harmful; in that example it learned 13% faster than the offline choice under normalized steps but stayed slower than a critic without privileged information, and it was not faster under plain SGD, whereas the exact variance-optimal baseline made the same signal useful.