- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies which privileged signals should be provided to the critic in asymmetric actor-critic methods and whether these inputs should adapt during training. The authors show that improving the critic’s return prediction does not necessarily benefit actor learning: a previously useful signal can increase update noise as the policy changes. The analysis decomposes a signal’s effect into a shared value offset and a change in relative action values, and examines their different effects on Monte Carlo and one-step TD updates. Based on this analysis, OIAC selects critic inputs online using an update-noise statistic. Experiments in small synthetic environments show that it can remove signals that become harmful during learning. However, noise reduction does not consistently accelerate learning, and benefits depend on the optimizer and baseline design. OIAC’s learning experiments are limited to tabular models with Monte Carlo updates.
Critical strengths
-
The problem is important and the mechanism analysis is clear. The paper distinguishes accurate return prediction by the critic from useful actor updates. Its offset–contrast decomposition explains why the effect of privileged information changes with the policy and advantage estimator, providing concrete insight into when privileged information helps or hurts asymmetric actor-critic learning.
-
Controlled experiments support the analysis, and the findings are carefully qualified. The authors use exact calculations and controlled experiments to examine the mechanism, while explicitly reporting that noise reduction need not accelerate learning, benefits depend on the optimizer, and harmful signals may be detected late. This helps distinguish evidence for the mechanism from evidence of practical optimization gains.
Critical weaknesses
-
OIAC’s practical advantage over simpler alternatives remains insufficiently established. In the main one-decision experiment, OIAC requires approximately 13% fewer episodes than IAAC under normalized updates, but approximately 8% more than the history-only critic. It shows no detectable speed-up under plain SGD, and the small Adam gain does not replicate on the confirmation seeds. Pooled estimates of the variance-optimal baseline also achieve substantial gains. These results leave the independent value of OIAC’s additional selection complexity unclear relative to these alternatives.
-
The algorithm does not yet address an important setting motivating the paper. Although the analysis emphasizes that the same signal can have opposite effects on Monte Carlo and TD updates, OIAC’s selection statistic and learning experiments remain restricted to Monte Carlo. Its applicability to bootstrapped actor-critic methods is therefore unestablished, and the experiments are confined to small synthetic environments and tabular models. While the insights built on established baseline-variance theory and advantage decompositions are informative, I find the incremental theoretical contribution insufficient to offset the current limitations in algorithmic scope and demonstrated benefit.
Questions for authors
-
When does online signal selection offer a practical advantage?
With comparable data access, estimation protocols, tuning effort, and computational budgets, under what conditions, if any, does OIAC outperform a history-only critic or a pooled estimate of the variance-optimal baseline ? Which existing results support such an advantage? -
How does the selection criterion extend to shared-parameter actors and bootstrapped updates?
For shared-parameter actors, the current statistic omits cross-step covariance terms; with TD/GAE, learned critics can also introduce bootstrap-induced bias. How would these factors affect the appropriate selection criterion and the ranking of candidate signals? What conditions or evidence would support reliable rankings beyond the current tabular Monte Carlo setting? -
What is the strongest practical evidence for adopting OIAC?
Given that the Adam speed-up did not replicate on the confirmation seeds, which demonstrated benefits provide the main practical justification for adopting OIAC, beyond the observed reduction in update noise?
General Comment to Reviewers
Dear Reviewers,
Thank you all for the time and thoughtful feedback you have devoted to our paper. We have done our best to address each of your comments in our responses and in the revised manuscript. In particular, following your suggestions, the latest revision frames the paper as an analysis with OIAC as a minimal test and clearly states the scope and limitations of the evidence in the abstract.
We understand how busy your schedules are, but as the discussion period is closing soon, we would greatly appreciate it if you could take a moment to review our responses and the latest revision. We remain happy to answer any remaining questions.
Best regards, The Authors