- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies policy-conditioned critics for off-policy actor–critic learning. It develops convergence guarantees under partial feature coverage and evolving behaviour policies, analyses replay sampling, and proposes functional-critic ensembles for exploration. A neural implementation is evaluated on eight continuous-control tasks and achieves strong empirical results.
Critical strengths
(S1: Potentially substantial theoretical contribution) Although stability and exploration with functional critics have precedents, the particular convergence guarantees may constitute a substantial theoretical advance. The combination of target-based TD analysis, partial feature coverage and policy-gradient generalisation appears especially valuable.
(S2: Stronger empirical comparisons) The experiments cover the full Dog and Humanoid task series and include strengthened SAC+ and TD3+ baselines. Matching critic counts and nominal critic-to-actor update ratios makes the competitive results more informative. Using one hyperparameter configuration across all eight tasks is also encouraging.
Critical weaknesses
(W1. Practical advance over earlier methods remains unclear) The paper appropriately credits Faccio et al. and Tang et al. and explains its methodological differences. However, it reports neither direct experimental comparisons with these methods nor ablations isolating the proposed representation and cross-policy training. I cannot tell how much these changes contribute to the reported gains. A comparison with PAVF (i.e. Faccio et al.) would help clarify the practical advancements.
(W2. Missing ensemble-size ablation and unmatched computational costs) The choice of 10 critics and 10 actors is not justified, and not enough ablated. Matching 10 critics in SAC+ and TD3+ strengthens the comparison, but does not really establish why many (and how many) functional critics are needed. Actor and critic counts should also be varied independently to distinguish their contributions.
(W3. Exploration mechanism is insufficiently demonstrated) For a fixed current actor and critic ensemble, uniformly sampling a critic gives the ensemble-average gradient in expectation. This alone does not establish the sustained exploration suggested by the posterior-sampling analogy. Actor diversity may help, but the experiments do not isolate its contribution or substantiate the uncertainty-based explanation. One experiment might help understanding is comparing against ensemble-mean gradients or less frequent critic resampling.
Questions for authors
Q is for questions and C is for comments.
(Q1: Addresses W1) Could you compare against PAVF under comparable training conditions and ablate the policy representation and cross-policy training? This would help identify the practical contribution beyond earlier methods.
(Q2: Addresses W2) I do not yet see why ten actors and ten critics are needed. Would, for example, two or five critics suffice with the actor count fixed? Please vary actor counts separately and report computational cost alongside sample efficiency. If the main benefit of the critic ensemble is diversity, could fewer functional critics with dropout, inspired by DroQ, achieve similar results?
(Q3: Addresses W3) What practical exploration benefit comes specifically from resampling a critic at every actor update? With ensemble sizes and interaction budgets fixed, could you maybe compare this with averaging the critics’ gradients and/or retaining one critic per actor for several updates?
(Q4: Discussion point) The paper motivates functional critics through reuse across changing policies and exploration across policy space. Under what conditions are these benefits likely to outweigh the additional approximation and computational costs? Identifying settings where policy conditioning helps, and where conventional critics suffice, would clarify the approach’s practical scope.
(C1) Several equation references contain duplicated labels: “Eq. equation 25” on p. 14, similar instances on p. 16, and “Equation equation 72” on p. 37. Please standardise these.
(C2) Appendix C.2, p. 19, states twice in consecutive paragraphs that the proposed method uses ten critics and a 10:1 critic-to-actor update ratio. Can remove one instance.
3 comments
Thank you for the detailed response and additional experiments. The new comparisons and ablations substantially address W1, and I appreciate the computational-cost reporting.
My remaining concern is the practical performance-cost tradeoff. A comparison of the proposed method using two critics against standard two-critic SAC and TD3 would be greatly helpful.
I also have two smal clarification questions (I apologize for raising these questions at this stage):
- Critic aggregation: How do SAC+ and TD3+ combine their ten critics for TD targets and actor updates? Taking the minimum over all ten could introduce greater pessimism than taking the minimum over two.
- Update ratio: In the updated paper and your main comments, Ours/Ours-NoCross use 11 critic updates per actor update, whereas the main text and Table 3 report 10:1. Appendix J uses 10:1 for timing. Could you clarify why this difference?
Overall, the rebuttal strengthens the paper, and I continue to recognise the potential theoretical contribution. I'm a little bit hesitant to raise my score, but these clarifications (and the two-critic comparison) would help resolve my remaining practical concerns.
Clarifications on critic aggregation and ensemble size
Thank you for carefully reading our rebuttal and for the thoughtful follow-up discussion. We appreciate your continued recognition of the potential theoretical contribution, your assessment that the new comparisons and ablations substantially address W1, and your acknowledgment of the computational-cost reporting.
Critic aggregation. Neither SAC+ nor TD3+ uses min-over-ten or min-over-two in our experiments. For each critic update, we uniformly sample one of the ten target critics to provide the bootstrap Q-value. Actor updates use the mean of all ten online critics, with the entropy term retained for SAC+. This follows RLPD's target-subset and actor-aggregation design (Ball et al., 2023, Algorithm 1), which allows the target-subset size to be either 1 or 2. We selected because it performed better than during baseline tuning. Thus, the ten-critic ensemble does not introduce the additional pessimism from minimizing over ten Q-values raised in your question. We have made these target-selection and actor-aggregation rules explicit in Appendix C.2 (“Baseline Construction”) of the revised manuscript.
Ensemble motivation and baseline design. Our functional-critic ensemble is motivated by the connection to PSRL in Section 4: each critic represents a candidate return functional over policy space, and actor updates use gradients from sampled functional critics. The ensemble is therefore part of our posterior-sampling-style exploration design, although this motivation does not prescribe a particular ensemble size.
We fixed the ensemble size at ten to match SAC+ and TD3+, which adopt the ten-critic default used in RLPD. Specifically, SAC+ adapts RLPD's critic-ensemble, layer-normalization, and high-update-ratio recipe to purely online off-policy learning without a prior offline dataset, with additional baseline tuning; TD3+ incorporates the same recipe. This reflects an established practical approach to sample-efficient off-policy RL: using suitable regularization to enable high update-to-data ratios (Chen et al., 2021; Li et al., 2023; Smith et al., 2022; Ball et al., 2023). Sergey Levine also summarizes this recipe in his public lecture Making Real-World Reinforcement Learning Practical, highlighting "good regularization to enable many gradient steps" (slide 9, 7:53–8:38). He later reiterates this high-UTD recipe and notes that the RLPD methodology also applies to training from scratch (15:30–16:22). Figure 2 and Appendix I document the benefits of these strengthened baselines across much of our benchmark. Our comparison therefore evaluates whether functional critics provide benefits within this effective training regime, with the additional computational overhead explicitly reported in Appendix J.
Two-critic comparison. Figure 2 already includes standard two-critic SAC and TD3. Your suggested comparison additionally requires a two-critic version of our method, which we have not evaluated. We agree that this would help characterize the performance-cost tradeoff at a smaller ensemble size. Unfortunately, the remaining rebuttal period does not allow us to complete a reliable multi-seed comparison. Our current results support the effectiveness of the evaluated configuration, but do not establish that ten critics are necessary or optimal, or that our method is superior at matched total computational cost.
Update ratio. We apologize for the confusion. We previously messed up the critic-to-actor update ratio (which is 10) and the update-to-data (UTD) ratio (which is 11) in our overall response and revised appendix. Already fixed.
We hope these clarifications address your questions and provide a clearer basis for assessing the empirical effectiveness of our approach. Thank you again for your careful assessment and constructive discussion.
Response to Reviewer 8BZZ
Thank you for recognizing the potential theoretical contribution of our analysis and the value of the strengthened baselines and shared hyperparameter configuration. We address your questions below.
W1 & Q1: Component ablations and comparison with PAVF.
New Section 5.2 directly addresses these questions. Figure 3 compares Ours with RawParam and ActionFingerprint while retaining the training procedure. Ours achieves higher final mean returns on all three Humanoid tasks, while RawParam retains much of its performance. Figure 4(d–f) examines Ours-NoCross, which pairs each replay transition with one uniformly sampled current actor. It retains much of Ours’ performance, showing that exhaustive actor–transition pairing within each update is not required for much of the observed gain. It still uses current policies for both critic conditioning and TD bootstrapping, with training spanning current policies over successive updates.
Crucially, our approach differs from PAVF in more than its encoder: a key distinction is which policies the critic learns to evaluate and provide gradients for. Ours and RawParam use current policies for both critic conditioning and TD bootstrapping, whereas historical-policy PAVF uses each transition’s collection-time policy. This training choice is motivated by our off-policy evaluation and actor–critic analysis. Figure 5 compares Ours and RawParam with a modernized historical-policy PAVF implementation on Humanoid Walk. Over the observed interval (approximately 83% of the planned budget), PAVF shows little learning, while both current-policy variants achieve substantially higher returns at the same interaction budget. See “Ours vs PAVF” in our overall response (2/2) for details.
W2 & Q2: Ensemble size and computational cost.
Ten actors and ten critics constitute one fixed configuration used across all eight tasks, rather than a theoretically required ensemble size. Our results demonstrate the viability of this configuration but do not establish that either count is necessary or optimal. Resource constraints during the rebuttal period prevented us from completing independent actor- and critic-count sweeps. Whether two or five critics, or fewer functional critics with DroQ-style dropout, can preserve the observed performance remains an open empirical question.
To examine critic-supervision volume, Fig. 4 compares Ours with SAC+/TD3+ using 10× larger critic minibatches, and Ours-NoCross with the standard-batch baselines. These controls match nominal TD-example counts per critic update and retain advantages in environment-sample efficiency in the learning curves. Appendix J (Table 5) now reports controlled single-GPU timings and model sizes, with details summarized in our updated overall response as well.
W3 & Q3: Exploration and critic resampling.
We agree that, for a fixed actor and critic ensemble, uniformly sampling a critic gives the ensemble-average gradient in expectation. Equality of conditional mean gradients does not imply identical optimization or data-collection trajectories, because sampled updates retain critic-dependent variability. However, this observation alone does not establish sustained or uncertainty-driven exploration. We have not completed the requested comparisons with ensemble-mean gradients or persistent critic assignments, so the present experiments do not isolate the benefit of resampling at every actor update.
The contribution of Section 4 is the conceptual connection: a sampled MDP induces a return functional over policy space, and a functional critic directly represents such a policy-conditioned object. This motivates our exploration design without providing a calibrated Bayesian posterior or a PSRL-style guarantee. The convergence guarantees in Section 3 are separate from this conceptual construction. Our empirical evidence concerns the complete algorithm; establishing the specific exploration benefit of critic resampling remains future work.
Q4: Practical scope.
Under our linear assumptions, useful reuse depends on coverage of target-relevant feature directions and reliable policy-gradient generalization across changing policies. This suggests that functional critics are promising when several related policies can share useful value information and saved environment interactions justify the additional computation. Conversely, conventional critics may suffice when little cross-policy reuse is needed, or may be preferable when policy-conditioned approximation is unreliable or wall-clock cost dominates. These are conditional expectations, not an empirically established characterization. Our benchmarks support environment-sample efficiency in the tested settings; they do not establish a universal computational advantage.
C1 & C2: Presentation.
Thank you for identifying these issues. We have standardized the equation references and removed the repeated ensemble-size/update-ratio statement in Appendix C.2.