acceptodds
Under review as a conference paper at ICLR 2027

Functional Critics for Off-Policy Actor-Critic: From Stability to Exploration

Abstract

Actor-critic methods are central to modern reinforcement learning, yet off-policy actor-critic theory remains far removed from practical algorithms, due in part to the interaction between the deadly triad and the continually changing target policy. We revisit functional critics, i.e., value functions that explicitly condition on the policy, and show that this representation provides a shared foundation for off-policy training stability and policy-space exploration. We first develop a convergent off-policy actor-critic framework under linear function approximation that accommodates target-based TD, partial coverage, and evolving behavior policies. Our analysis combines a policy-evaluation guarantee under partial coverage with policy-gradient generalization of functional critics to yield a dual trust-coverage framework that controls policy-gradient error under evolving behavior policies while enabling greater reuse of off-policy data. We further develop a replay-buffer analysis under i.i.d. sampling, yielding a principled replay-sampling design for minimizing the replay-sample complexity required to reach a prescribed evaluation accuracy. We next show that functional critics provide a natural representation for model-free posterior-sampling-style exploration. Standard critic ensembles do not naturally represent joint uncertainty over return functions across policy space, whereas functional-critic ensembles represent policy-conditioned return functions directly. This motivates a particle-based, model-free analogue of posterior-sampling-style exploration without explicit model learning. Finally, we instantiate these ideas in a deep functional actor-critic algorithm that, despite omitting several standard off-policy AC heuristics, achieves strong performance across the full Dog and Humanoid task series of the DeepMind Control Suite.

Primary area:
reinforcement learning
Keywords:
off-policy RL, actor-critic

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_8BZZOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies policy-conditioned critics for off-policy actor–critic learning. It develops convergence guarantees under partial feature coverage and evolving behaviour policies, analyses replay sampling, and proposes functional-critic ensembles for exploration. A neural implementation is evaluated on eight continuous-control tasks and achieves strong empirical results.

Critical strengths

(S1: Potentially substantial theoretical contribution) Although stability and exploration with functional critics have precedents, the particular convergence guarantees may constitute a substantial theoretical advance. The combination of target-based TD analysis, partial feature coverage and policy-gradient generalisation appears especially valuable.

(S2: Stronger empirical comparisons) The experiments cover the full Dog and Humanoid task series and include strengthened SAC+ and TD3+ baselines. Matching critic counts and nominal critic-to-actor update ratios makes the competitive results more informative. Using one hyperparameter configuration across all eight tasks is also encouraging.

Critical weaknesses

(W1. Practical advance over earlier methods remains unclear) The paper appropriately credits Faccio et al. and Tang et al. and explains its methodological differences. However, it reports neither direct experimental comparisons with these methods nor ablations isolating the proposed representation and cross-policy training. I cannot tell how much these changes contribute to the reported gains. A comparison with PAVF (i.e. Faccio et al.) would help clarify the practical advancements.

(W2. Missing ensemble-size ablation and unmatched computational costs) The choice of 10 critics and 10 actors is not justified, and not enough ablated. Matching 10 critics in SAC+ and TD3+ strengthens the comparison, but does not really establish why many (and how many) functional critics are needed. Actor and critic counts should also be varied independently to distinguish their contributions.

(W3. Exploration mechanism is insufficiently demonstrated) For a fixed current actor and critic ensemble, uniformly sampling a critic gives the ensemble-average gradient in expectation. This alone does not establish the sustained exploration suggested by the posterior-sampling analogy. Actor diversity may help, but the experiments do not isolate its contribution or substantiate the uncertainty-based explanation. One experiment might help understanding is comparing against ensemble-mean gradients or less frequent critic resampling.

Questions for authors

Q is for questions and C is for comments.

(Q1: Addresses W1) Could you compare against PAVF under comparable training conditions and ablate the policy representation and cross-policy training? This would help identify the practical contribution beyond earlier methods.

(Q2: Addresses W2) I do not yet see why ten actors and ten critics are needed. Would, for example, two or five critics suffice with the actor count fixed? Please vary actor counts separately and report computational cost alongside sample efficiency. If the main benefit of the critic ensemble is diversity, could fewer functional critics with dropout, inspired by DroQ, achieve similar results?

(Q3: Addresses W3) What practical exploration benefit comes specifically from resampling a critic at every actor update? With ensemble sizes and interaction budgets fixed, could you maybe compare this with averaging the critics’ gradients and/or retaining one critic per actor for several updates?

(Q4: Discussion point) The paper motivates functional critics through reuse across changing policies and exploration across policy space. Under what conditions are these benefits likely to outweigh the additional approximation and computational costs? Identifying settings where policy conditioning helps, and where conventional critics suffice, would clarify the approach’s practical scope.

(C1) Several equation references contain duplicated labels: “Eq. equation 25” on p. 14, similar instances on p. 16, and “Equation equation 72” on p. 37. Please standardise these.

(C2) Appendix C.2, p. 19, states twice in consecutive paragraphs that the proposed method uses ten critics and a 10:1 critic-to-actor update ratio. Can remove one instance.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8BZZ

Thank you for recognizing the potential theoretical contribution of our analysis and the value of the strengthened baselines and shared hyperparameter configuration. We address your questions below.

W1 & Q1: Component ablations and comparison with PAVF.

New Section 5.2 directly addresses these questions. Figure 3 compares Ours with RawParam and ActionFingerprint while retaining the training procedure. Ours achieves higher final mean returns on all three Humanoid tasks, while RawParam retains much of its performance. Figure 4(d–f) examines Ours-NoCross, which pairs each replay transition with one uniformly sampled current actor. It retains much of Ours’ performance, showing that exhaustive actor–transition pairing within each update is not required for much of the observed gain. It still uses current policies for both critic conditioning and TD bootstrapping, with training spanning current policies over successive updates.

Crucially, our approach differs from PAVF in more than its encoder: a key distinction is which policies the critic learns to evaluate and provide gradients for. Ours and RawParam use current policies for both critic conditioning and TD bootstrapping, whereas historical-policy PAVF uses each transition’s collection-time policy. This training choice is motivated by our off-policy evaluation and actor–critic analysis. Figure 5 compares Ours and RawParam with a modernized historical-policy PAVF implementation on Humanoid Walk. Over the observed interval (approximately 83% of the planned budget), PAVF shows little learning, while both current-policy variants achieve substantially higher returns at the same interaction budget. See “Ours vs PAVF” in our overall response (2/2) for details.

W2 & Q2: Ensemble size and computational cost.

Ten actors and ten critics constitute one fixed configuration used across all eight tasks, rather than a theoretically required ensemble size. Our results demonstrate the viability of this configuration but do not establish that either count is necessary or optimal. Resource constraints during the rebuttal period prevented us from completing independent actor- and critic-count sweeps. Whether two or five critics, or fewer functional critics with DroQ-style dropout, can preserve the observed performance remains an open empirical question.

To examine critic-supervision volume, Fig. 4 compares Ours with SAC+/TD3+ using 10× larger critic minibatches, and Ours-NoCross with the standard-batch baselines. These controls match nominal TD-example counts per critic update and retain advantages in environment-sample efficiency in the learning curves. Appendix J (Table 5) now reports controlled single-GPU timings and model sizes, with details summarized in our updated overall response as well.

W3 & Q3: Exploration and critic resampling.

We agree that, for a fixed actor and critic ensemble, uniformly sampling a critic gives the ensemble-average gradient in expectation. Equality of conditional mean gradients does not imply identical optimization or data-collection trajectories, because sampled updates retain critic-dependent variability. However, this observation alone does not establish sustained or uncertainty-driven exploration. We have not completed the requested comparisons with ensemble-mean gradients or persistent critic assignments, so the present experiments do not isolate the benefit of resampling at every actor update.

The contribution of Section 4 is the conceptual connection: a sampled MDP induces a return functional over policy space, and a functional critic directly represents such a policy-conditioned object. This motivates our exploration design without providing a calibrated Bayesian posterior or a PSRL-style guarantee. The convergence guarantees in Section 3 are separate from this conceptual construction. Our empirical evidence concerns the complete algorithm; establishing the specific exploration benefit of critic resampling remains future work.

Q4: Practical scope.

Under our linear assumptions, useful reuse depends on coverage of target-relevant feature directions and reliable policy-gradient generalization across changing policies. This suggests that functional critics are promising when several related policies can share useful value information and saved environment interactions justify the additional computation. Conversely, conventional critics may suffice when little cross-policy reuse is needed, or may be preferable when policy-conditioned approximation is unreliable or wall-clock cost dominates. These are conditional expectations, not an empirically established characterization. Our benchmarks support environment-sample efficiency in the tested settings; they do not establish a universal computational advantage.

C1 & C2: Presentation.

Thank you for identifying these issues. We have standardized the equation references and removed the repeated ensemble-size/update-ratio statement in Appendix C.2.

Thank you for the detailed response and additional experiments. The new comparisons and ablations substantially address W1, and I appreciate the computational-cost reporting.

My remaining concern is the practical performance-cost tradeoff. A comparison of the proposed method using two critics against standard two-critic SAC and TD3 would be greatly helpful.

I also have two smal clarification questions (I apologize for raising these questions at this stage):

  • Critic aggregation: How do SAC+ and TD3+ combine their ten critics for TD targets and actor updates? Taking the minimum over all ten could introduce greater pessimism than taking the minimum over two.
  • Update ratio: In the updated paper and your main comments, Ours/Ours-NoCross use 11 critic updates per actor update, whereas the main text and Table 3 report 10:1. Appendix J uses 10:1 for timing. Could you clarify why this difference?

Overall, the rebuttal strengthens the paper, and I continue to recognise the potential theoretical contribution. I'm a little bit hesitant to raise my score, but these clarifications (and the two-critic comparison) would help resolve my remaining practical concerns.

AuthorsOpenReview ↗

Clarifications on critic aggregation and ensemble size

Thank you for carefully reading our rebuttal and for the thoughtful follow-up discussion. We appreciate your continued recognition of the potential theoretical contribution, your assessment that the new comparisons and ablations substantially address W1, and your acknowledgment of the computational-cost reporting.

Critic aggregation. Neither SAC+ nor TD3+ uses min-over-ten or min-over-two in our experiments. For each critic update, we uniformly sample one of the ten target critics to provide the bootstrap Q-value. Actor updates use the mean of all ten online critics, with the entropy term retained for SAC+. This follows RLPD's target-subset and actor-aggregation design (Ball et al., 2023, Algorithm 1), which allows the target-subset size ZZ to be either 1 or 2. We selected Z=1Z=1 because it performed better than Z=2Z=2 during baseline tuning. Thus, the ten-critic ensemble does not introduce the additional pessimism from minimizing over ten Q-values raised in your question. We have made these target-selection and actor-aggregation rules explicit in Appendix C.2 (“Baseline Construction”) of the revised manuscript.

Ensemble motivation and baseline design. Our functional-critic ensemble is motivated by the connection to PSRL in Section 4: each critic represents a candidate return functional over policy space, and actor updates use gradients from sampled functional critics. The ensemble is therefore part of our posterior-sampling-style exploration design, although this motivation does not prescribe a particular ensemble size.

We fixed the ensemble size at ten to match SAC+ and TD3+, which adopt the ten-critic default used in RLPD. Specifically, SAC+ adapts RLPD's critic-ensemble, layer-normalization, and high-update-ratio recipe to purely online off-policy learning without a prior offline dataset, with additional baseline tuning; TD3+ incorporates the same recipe. This reflects an established practical approach to sample-efficient off-policy RL: using suitable regularization to enable high update-to-data ratios (Chen et al., 2021; Li et al., 2023; Smith et al., 2022; Ball et al., 2023). Sergey Levine also summarizes this recipe in his public lecture Making Real-World Reinforcement Learning Practical, highlighting "good regularization to enable many gradient steps" (slide 9, 7:53–8:38). He later reiterates this high-UTD recipe and notes that the RLPD methodology also applies to training from scratch (15:30–16:22). Figure 2 and Appendix I document the benefits of these strengthened baselines across much of our benchmark. Our comparison therefore evaluates whether functional critics provide benefits within this effective training regime, with the additional computational overhead explicitly reported in Appendix J.

Two-critic comparison. Figure 2 already includes standard two-critic SAC and TD3. Your suggested comparison additionally requires a two-critic version of our method, which we have not evaluated. We agree that this would help characterize the performance-cost tradeoff at a smaller ensemble size. Unfortunately, the remaining rebuttal period does not allow us to complete a reliable multi-seed comparison. Our current results support the effectiveness of the evaluated configuration, but do not establish that ten critics are necessary or optimal, or that our method is superior at matched total computational cost.

Update ratio. We apologize for the confusion. We previously messed up the critic-to-actor update ratio (which is 10) and the update-to-data (UTD) ratio (which is 11) in our overall response and revised appendix. Already fixed.

We hope these clarifications address your questions and provide a clearer basis for assessing the empirical effectiveness of our approach. Thank you again for your careful assessment and constructive discussion.

Reviewer_6tNrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies policy-conditioned critics as a common representation for off-policy actor-critic stability and exploration. It derives evaluation and convergence results under linear functional approximation, proposes trust metrics for coverage and critic reuse, and analyzes replay sampling. A neural functional-critic ensemble is evaluated on the Dog and Humanoid series of DeepMind Control.

Critical strengths

The distinction between feature coverage for policy evaluation and gradient generalization across policies is conceptually useful. The replay analysis gives a structured view of data reuse.

Critical weaknesses

he convergence result requires a common linear coefficient vector representing Q for every policy, plus Bellman completeness and gradient compatibility. These strong conditions are not shown to hold for the neural system. The trust metrics used to trigger behavior-policy refresh and critic reevaluation depend on target-policy quantities that are unknown in practice. Their estimation and resulting decision errors are not analyzed. The exploration method is posterior-sampling-inspired, without posterior sampling, bootstrap masks, or episode-level sampled hypotheses. There is no direct comparison with strong exploration-specific baselines. The main experiments combine policy conditioning, a learned actor encoder, a transformer, actor and critic ensembles, and high-rate critic updates. Component ablations are insufficient, and the methods are not matched for total critic-side computation

Questions for authors
  1. How are the two trust metrics estimated in the neural implementation, and how sensitive are refresh decisions to estimation error?
  2. What are the separate effects of policy conditioning, the policy-input gradient, actor probes, and the ensemble?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 6tNr (1/2)

Our work contributes new off-policy actor–critic convergence guarantees under explicit linear assumptions and a theory-inspired neural algorithm with strong empirical performance. We clarify their significance and address the implementation and empirical concerns below.

Theoretical contribution, assumptions, and practical relevance.

Establishing convergence for target-based off-policy actor–critic is already a nontrivial theoretical problem. Barakat et al. (2022) establish target-based actor–critic convergence in the on-policy setting and explicitly identify the off-policy extension as future work. Our analysis addresses this off-policy problem while also accommodating partial feature coverage and evolving behavior policies, and separately develops replay-sampling guarantees under the stated i.i.d. assumptions. By bringing these practical mechanisms within a common framework, our results constitute a substantial contribution beyond the off-policy extension itself.

Our assumptions build on established assumption families in RL theory. The novelty and significance of our linear analysis should therefore be assessed against prior guarantees in comparable theoretical settings. Requiring these assumptions to be verified for the full neural implementation as a prerequisite for recognizing the theoretical contribution would impose a different research objective. A convergence guarantee for the full practical neural training system is a distinct, broader problem outside our scope; its absence does not negate the advances established here.

The theory also motivates concrete algorithmic choices. Our neural implementation follows the established approach of deriving principles in a tractable theoretical setting and testing their usefulness in practical systems. In particular, our analysis of off-policy evaluation and policy-gradient generalization motivates training critics on current target policies using replayed transitions. Historical-policy PAVF instead ties each transition's policy conditioning and TD bootstrap to its collection-time policy. RawParam preserves our theory-motivated training design and replaces only the encoder with PAVF-style full-parameter inputs; it is an ablation of our method, not a reproduction of historical-policy PAVF.

The new experiments support the practical value of this approach. RawParam retains much of Ours' performance (Fig. 3). Ours-NoCross also retains strong performance with the same nominal TD-example count per critic update as the standard SAC+ and TD3+ baselines (Fig. 4). At the common interim budget on Humanoid Walk, approximately 83% of the planned training budget, our modernized historical-policy PAVF implementation shows little learning while Ours and RawParam have already improved substantially (Fig. 5). Strong performance without clipped double-Q targets, target-policy smoothing, or entropy regularization further supports the practical value of our theory-inspired design.

Posterior-sampling-inspired exploration.

The contribution of Section 4 is the connection between posterior sampling over MDPs and distributions over policy-conditioned return functionals. Each functional critic models a return functional over policy space, enabling policy improvement against a sampled functional without explicitly maintaining and solving sampled MDP models. This motivates our exploration construction; it does not establish that the learned ensemble is a calibrated posterior. We do not use bootstrap masks. Algorithm 4 samples a rollout actor at the beginning of each episode, but critics are resampled during actor optimization and actor parameters continue to evolve, so this is not an episode-persistent posterior hypothesis. Our experiments establish the empirical viability of the complete algorithm against strong actor–critic baselines. We have not provided comparisons with exploration-specific baselines or experiments isolating the exploration benefit of critic resampling, and therefore do not claim PSRL-style exploration guarantees or an experimentally isolated advantage from that mechanism.

Total critic-side computation.

The added experiments directly address TD-supervision volume through two complementary controls: Ours versus SAC+/TD3+ with tenfold larger critic minibatches (Fig. 4a–c), and Ours-NoCross versus the standard-batch baselines (Fig. 4d–f). Both comparisons match nominal TD-example counts per critic update and retain environment-sample-efficiency advantages in the learning curves. These controls do not equate total computation. Appendix J (Table 5) now reports controlled single-GPU timings and model sizes under a common 10:1 update schedule; details are summarized in our updated overall response. Our empirical claim concerns environment-sample efficiency under the supervision controls, rather than superior performance at matched total computational cost.

AuthorsOpenReview ↗

Response to Reviewer 6tNr (2/2)

Q1: Trust-metric estimation, calibration, and sensitivity.

The base neural algorithm deliberately omits trust-triggered refreshes to preserve a simple training configuration across all eight tasks without introducing additional threshold tuning. Our theory uses prescribed trust criteria, but does not provide an automatic calibration rule for their learned-feature neural approximations. Appendix F (Fig. 8) showcases a targeted Dog Walk example in which evaluation-trust-guided reuse, with an empirically calibrated threshold and delayed activation, further improves environment-sample efficiency over the base algorithm. Adaptive selection of the neural trust thresholds is an important direction for future work. The distinction between introducing a useful mechanism and automating its calibration also has precedent in SAC: the original method required task-specific reward-scale tuning, while subsequent work introduced automatic temperature adjustment [1, 2].

Appendix F details the evaluation-trust estimator and its calibration, using frozen critic features and replay-buffer approximations of the relevant expectations.

Appendix F ("Heuristic Gradient-Guided Critic Reuse") also describes a separately implemented heuristic for the gradient-generalization criterion Δk,t\Delta_{k,t}. It uses normalized features from frozen critics and a ridge-regularized reference covariance estimated from replay minibatches. Action- and feature-Jacobian proxies approximate the two coefficients, with diagonal inverse-covariance weighting and rescaled feature-coordinate subsampling for the second. After averaging each coefficient over actor–critic pairs, we form Δ^k,t=max⁡C^1(k,t),C^2(k,t)\widehat{\Delta}_{k,t}=\max\\{\widehat{C}_1^{(k,t)},\widehat{C}_2^{(k,t)}\\}. At a scheduled return to critic training, the implementation extends the actor-update phase when this proxy does not exceed the prescribed threshold, subject to a cap on consecutive extensions; otherwise, critic training resumes. This extension has been implemented but is not yet evaluated in the reported experiments. These approximations do not preserve the theoretical gradient-error guarantee.

The reported evaluation-trust study in Appendix F demonstrates a potential empirical benefit; it does not quantify how estimation errors affect refresh decisions. A principled analysis of such decision errors remains open.

[1] Haarnoja et al. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. ICML 2018. https://proceedings.mlr.press/v80/haarnoja18b.html

[2] Haarnoja et al. Soft Actor-Critic Algorithms and Applications. https://arxiv.org/abs/1812.05905

Q2: Separate effects of policy conditioning, the policy-input gradient, actor probes, and the ensemble.

New Section 5.2 and Appendix D provide complementary controls for these components, with the following conclusions:

  • Policy conditioning. Actor-ID replaces the evolving policy representation with a learned embedding of actor identity (Appendix D, Fig. 6). Its performance is task-dependent: its final mean return is higher than Ours on Stand, close on Walk, and lower on Run. This replacement also removes the explicit policy-input gradient, so it is a joint representation-and-gradient control.

  • Policy-input gradient. Detached-policy-gradient retains the policy-conditioned critic and the action-input gradient, but removes the actor gradient through the explicit policy input. Ours achieves higher final mean returns on all three Humanoid tasks (Fig. 6), supporting an empirical benefit from retaining this gradient path in the tested setting.

  • Actor probes and representation. ActionFingerprint retains learned probes but uses only actor actions and an affine adapter; RawParam instead encodes the full actor parameter vector. Ours achieves higher final mean returns than both on all three Humanoid tasks, while RawParam retains much of its performance (Fig. 3). These results favor our representation while showing that strong performance is possible without our probe-based encoder. The comparisons evaluate complete encoding choices; they do not separately isolate probe learning, hidden activations, and attention.

  • Ensemble. The above controls retain the same actor and critic ensemble sizes, so their differences are not attributable to changing ensemble size. SAC+ and TD3+ also use ten critics, providing strong ensemble baselines. We have not completed independent actor- and critic-count sweeps within the available resources; the results therefore establish the effectiveness of the tested configuration, without establishing that ten actors or ten critics are necessary or optimal.

Reviewer_7TXuOpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is focusing on off-policy actor-critic methods, and specifically on the theory behind their training stability and exploration. In this context, a "functional critic" is a value functional that explicitly takes the policy as input. The paper uses functional critics to establish theory and analyze the stability and exploration of these algorithms.

The paper first presents an off-policy AC convergence result for linear function approximation. Then to address the exploration question, they show a structural connection between functional critics and PSRL. Then in the end, they present an algorithm based on their theoretical results which achieves strong performance in full Dog and Humanoid tasks from DMC. To my understanding, the algorithm and the results serve as an empirical evidence to suggest the theoretical results derived under linear function approximation transfers to realistic neural networks.

Critical strengths
  • The theoretical results are interesting. They show that: if the behavioural data cover the feature directions relevant to a policy, target network can evaluate that policy, even if the covariance is singular. This is a more informative criterion than requiring coverage of every state-action pair. The separation of exploration and training stability is also useful.

  • The empirical results and the neural implementation, although slightly disconnected from theory, demonstrates the viability of policy-conditioned critics on high scale DMC tasks. They show strong performance against SAC+ and TD3+.

  • The connection to PSRL, and how a critic that takes policy as input can model how returns vary across candidate policies, is a useful and interesting conceptual contribution.

Critical weaknesses
  1. PAVF already combines a policy-conditioned Q function, off-policy, replay, and both actor-gradient paths. The empirical results are comparing to SAC+ and TD3+, which show us that the full algorithmic setup is competitive. But it does not distinguish the individual effect of the novel parts of this method such as the encoder and cross-policy training, or the additional critic computation. PAVF (Faccio et al. 2021) and learned policy fingerprints (Faccio et al. 2022) already exist. So distinguishing the effect of the paper's unique contributions on performance is critical.

  2. There is a gap between the theorem and the experiments. The main algorithm uses replay states for the actor update, but the theorem uses the initial-state distribution.

Questions for authors
  1. I understand that PAVF is not directly comparable perhaps. But can you use Faccio et al. (2022)'s action-only fingerprinting instead of your proposed encoder for instance, while keeping everything same? This could give us an idea about the contributions in performance due to your encoder. Any other ablation narrowing down on the novel components would help the paper.

  2. Can you in paper clarify the difference between the theorem and the actor update? Just making this visible should be enough, I don't think there is need for new theory.

  3. As far as I can tell, since your critic uses multiple continuation policies, your critics are trained with more TD targets per sampled transition compared to SAC+ and TD3+. Please correct me if I am wrong here. If this is the case, can you have a more compute-matched comparison? Perhaps the methods can be compared in terms of wall-clock time. Otherwise, your method does more updates per sampled transition so comparison is difficult.

A controlled comparison showing a reproducible benefit from either the proposed encoder or cross-policy training at a comparable compute budget would make me reconsider my score. A clarification of the actor objective and the scope of the convergence result would address my second concern.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7TXu

Thank you for recognizing our theoretical results, empirical performance against strong SAC+ and TD3+ baselines, and conceptual connection between functional critics and PSRL.

W1 & Q1: Ablations and comparison with PAVF.

As requested, we added an action-only fingerprint variant inspired by Faccio et al. (2022), replacing our policy encoder while retaining the training procedure and TD-pairing schedule. Across four seeds on each of the three Humanoid tasks, Ours achieves higher final mean returns than ActionFingerprint and RawParam, while RawParam retains much of its performance (Fig. 3). We also examine reduced actor–transition pairing (Fig. 4d–f) and include Actor-ID and detached-policy-gradient controls in Appendix D (Fig. 6).

Crucially, our approach differs from PAVF in more than its encoder: a key distinction is which policies the critic learns to evaluate and provide gradients for. Ours and RawParam use current actors for both critic conditioning and TD bootstrapping, whereas historical-policy PAVF uses each transition’s collection-time policy. This training choice is motivated by our off-policy evaluation and actor–critic analysis; it is distinct from the pairing count examined by Ours-NoCross, which also retains current-policy supervision.

In Fig. 5, the modernized historical-policy PAVF implementation shows little learning on Humanoid Walk over the observed interval (approximately 63% of the planned budget), while both Ours and RawParam achieve substantially higher returns at the same environment-step budget. These results complement the encoder ablations and support the practical relevance of our theory-inspired training design beyond the encoder. See “Ours vs PAVF” in our overall response (2/2) for details.

W2 & Q2: The theoretical objective and the practical actor update.

You are correct that the theoretical analysis and the practical actor update use different state distributions. Our convergence analysis concerns Jρ(θ)=Es∼ρ[Vπθ(s)]J_\rho(\theta)=\mathbb E_{s\sim\rho}[V^{\pi_\theta}(s)], where ρ\rho is the initial-state distribution. Algorithm 4 instead samples actor-update states from replay and uses the surrogate

JdR(πθ;ξ)=Es∼dR[Q^(πθ,s,πθ(s);ξ)],\displaystyle J_{d_{\mathcal R}}(\pi_\theta;\xi) =\mathbb E_{s\sim d_{\mathcal R}} \left[\widehat Q(\pi_\theta,s,\pi_\theta(s);\xi)\right],

with the current replay sampling distribution and critic parameters held fixed during each actor update.

We have revised Appendix B.2 to formulate the deterministic functional gradient for a general fixed state distribution ν\nu and explicitly distinguish ν=ρ\nu=\rho from ν=dR\nu=d_{\mathcal R}. Both use the action-input and explicit-policy-input gradient paths, but the different state weighting generally changes the objective, even with an exact critic. The revised discussion therefore makes clear that our linear convergence analysis does not directly establish convergence of the replay-state neural implementation.

Q3: Critic-supervision mismatch.

You are correct that cross-policy training supplies more TD examples per sampled transition. We added two complementary controls: Ours versus SAC+/TD3+ with 10× larger critic minibatches (Fig. 4a–c), and Ours-NoCross versus the standard-batch baselines (Fig. 4d–f). Both comparisons match nominal TD-example counts per critic update and retain advantages in environment-sample efficiency in the learning curves. These supervision controls do not equate total computation. We have now added controlled single-GPU timings and model sizes in Appendix J (Table 5), quantifying the runtime overhead under a common 10:1 update schedule. Please see Section 5.2 and our updated overall response further details.

Thank you for the rebuttal. I have also checked your response to Reviewer MEEq which also answers some off my questions. Since I personally don't have any other objections to the paper, I am updating my score.

AuthorsOpenReview ↗

Thank you for your thoughtful feedback

Thank you for your careful consideration of our rebuttal and our response to Reviewer MEEq. We appreciate your constructive feedback and are glad that the revisions and clarifications have addressed your concerns.

Reviewer_MEEqOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies functional critics, Q(π,s,a)Q(\pi,s,a) that take the policy as input, as a shared basis for off-policy training stability and exploration. Under linear function approximation, it proves target-based TD evaluation under partial feature coverage (Theorem 1) and actor-critic convergence with evolving behavior policies through a dual trust-coverage rule (Theorem 2). An i.i.d. analysis gives a replay-sampling design, and critic ensembles motivate posterior-sampling-style exploration. The authors report competitive performance against SAC+ and TD3+ on DeepMind Control Dog and Humanoid tasks.

Critical strengths

The paper contributes a convergence analysis of off-policy actor-critic under linear function approximation that jointly handles target-based TD, feature-based coverage and evolving behavior policies, along with an i.i.d. replay-sampling design and a deep functional actor-critic with ensemble exploration. Its experiments show that this deep system, using one hyperparameter configuration and omitting clipped double-Q targets, exploration noise and entropy regularization, matches or exceeds tuned SAC+ and TD3+ in mean return on the DeepMind Control Dog and Humanoid tasks, which demonstrates the practical viability of neural functional critics. The authors introduce the functional-gradient operator GπG_\pi (Eq. 8), which makes the gradient error linear in the critic error, and they differentiate JρJ_\rho directly, avoiding the surrogate objective. On the theory side, Theorem 3 proves finite-time convergence of the inner TD loop under Markovian noise and a singular behavior covariance, which yields the evaluation guarantee of Theorem 1. Theorem 2 carries this to the full actor-critic, bounding the average squared gradient norm by O(1/(ηM))O(1/(\eta M)) plus an error floor O~(cgϵˉ2)\tilde{O}(c_g \bar\epsilon^2).

Critical weaknesses

My concerns are with a hidden scaling factor, incorrect constants in the proof of Theorem 1, and the choice of baselines. The proposed bounds hold with the corrections below.

(1) K≍ϵ−2K \asymp \epsilon^{-2} in Theorem 1 hides a strong dependence on λ+=λmin⁡+(A~π)\lambda^+ = \lambda_{\min}^+(\tilde A_\pi). Eq. 60 gives K≈log⁡(⋅)/(2αλ+)K \approx \log(\cdot) / (2\alpha\lambda^+), so 1+αK1 + \alpha K is of order 1/λ+1/\lambda^+. The second condition in Remark 4, which is the noise term of Lemma 8 (Eq. 66), then forces α∝ϵ2(1−γ)2(λ+)2/τα\alpha \propto \epsilon^2 (1-\gamma)^2 (\lambda^+)^2 / \tau_\alpha, a dependence that the "∝\propto" in Eq. 59 hides. Hence K≈ταϵ−2(1−γ)−2(λ+)−3K \approx \tau_\alpha \epsilon^{-2} (1-\gamma)^{-2} (\lambda^+)^{-3} up to logarithms.

(2) Two steps in the proof of Theorem 1 have incorrect constants. Let E=EapproxE = E_{\mathrm{approx}}, vk=Q^k−Q⋆v_k = \hat Q_k - Q^\star, x=Pπ∣vk−1∣x = P_\pi \lvert v_{k-1} \rvert, b=∣Q^k−Φξ⋆(wk−1)∣b = \lvert \hat Q_k - \Phi\xi^\star(w_{k-1}) \rvert and c=∣Φ(ξ′−ξ⋆(wk−1))∣c = \lvert \Phi(\xi' - \xi^\star(w_{k-1})) \rvert, so Eq. 61 reads ∣vk∣≤γx+b+E+c\lvert v_k \rvert \le \gamma x + b + E + c.

  • Eq. 61 to Eq. 62. With y=b+E+cy = b + E + c, the inequality (γx+y)2≤γx2+y2/(1−γ)(\gamma x + y)^2 \le \gamma x^2 + y^2/(1-\gamma) together with y2≤3(b2+E2+c2)y^2 \le 3(b^2 + E^2 + c^2) gives the coefficient 3/(1−γ)3/(1-\gamma) on each of b2b^2, E2E^2 and c2c^2. Eq. 62 instead has 44 on E2E^2 and c2c^2, which is missing the factor 1/(1−γ)1/(1-\gamma).
  • Eq. 62 to Eq. 65. Unrolling the recursion adds a factor 1/(1−γ)1/(1-\gamma) from the geometric sum, so the bound in L2(dπ)L^2(d_\pi) should be E∥vT∥2≤γTE∥v0∥2+3(1−γ)−2[C(π,μ)(ϵ2+E2)+E2]\mathbb{E}\lVert v_T \rVert^2 \le \gamma^T \mathbb{E}\lVert v_0 \rVert^2 + 3(1-\gamma)^{-2} \left[ C(\pi,\mu)(\epsilon^2 + E^2) + E^2 \right]. The 1/(1−γ)1/(1-\gamma) and the bare E2E^2 in Eq. 65 do not follow, and even Eq. 62 as written gives 16/(1−γ)316/(1-\gamma)^3 on the ϵ2\epsilon^2 term.

Eq. 70 has the correct order and Theorem 2 relies on it, so these corrections change the proof of Theorem 1 but do not affect Theorem 2. Separately, the O~\tilde{O} in Eq. 12 hides the factor (1−γ)−2(1-\gamma)^{-2} that appears in Eq. 71.

(3) I could find three typos in the proofs: In Eq. 61, wK−1w_{K-1} should be wk−1w_{k-1}. Lemma 2 and the bound on (T3)+(T4) cite "Assumption 2" where they mean Condition 2 of Theorem 3. In Appendix G.1, w0w_0 should be uu.

(4) The main experiments compare only against in-house SAC+ and TD3+, neither of which has a policy-conditioned critic, so they do not show what the functional critic contributes. The closest prior method is PAVF (Faccio et al.), which the paper itself calls the closest formal antecedent of its gradient (line 685). PAVF is also trained by TD and uses both a DPG term and a derivative through the policy input as per Table 1. The paper excludes it, along with other earlier policy-conditioned methods, based on a thesis benchmark on Pendulum and HalfCheetah and on unreported preliminary attempts (lines 1000–1002). To my knowledge PAVF has no published Dog or Humanoid results, so it would need to be run on these tasks within the same training recipe. The authors also acknowledge that critic-side computation is not matched (lines 455–457), and no wall-clock time is reported, so sample efficiency cannot be weighed against the added cost.

Francesco Faccio, Louis Kirsch, and Jürgen Schmidhuber. Parameter-based value functions. In International Conference on Learning Representations (ICLR), 2021.

Questions for authors

Q1. As PAVF is the closest antecedent to your gradient, can you add a PAVF-style variant within your own training recipe on the same tasks, replacing your policy encoder with raw parameters (using a smaller actor if needed)?

Q2. Can you report wall-clock time per environment step for your method and each baseline, given that critic-side computation is not matched?

Q3. Can you revise the theory section to state KK in Theorem 1 explicitly in terms of λmin⁡+(A~π)\lambda_{\min}^+(\tilde A_\pi), τα\tau_\alpha and 1−γ1-\gamma, correct the constants in Eqs. 62 and 65, and show the resulting horizon factors in the bounds of Theorems 1 and 2?

5 comments
AuthorsOpenReview ↗

Response to Reviewer MEEq

Thank you for recognizing the contribution of our theoretical results and the practical viability of our implementation.

W1 & Q3: Explicit dependence on problem-dependent factors in the theoretical claims.

Thank you for raising this point. We have now made the dependence on these factors (including 1−γ1-\gamma, the mixing time τα\tau_\alpha, and c0=λmin⁡+(A~π)c_0=\lambda_{\min}^{+}(\widetilde{\mathbf{A}}_\pi)) explicit in the statements of the theoretical results (including Theorem 1 and Remark 4) and other related parts of the derivation (e.g., Eqs. 59 and 60) in the revised draft. More precisely, the choice of KK is now

K=O~(ταϵ2(1−γ)2c03),\displaystyle K=\widetilde{O}\left(\frac{\tau_\alpha}{\epsilon^2(1-\gamma)^2c_0^3}\right),

as suggested by the reviewer.

W2 & Q3: Corrections to Eqs. 62 and 65 and explicit horizon dependence.

We have corrected both steps in Appendix G.5. The squared-error recursion in Eq. (62) now has coefficient 3/(1−γ)3/(1-\gamma) on each of the three error terms. Summing the recursion gives the factor 3/(1−γ)23/(1-\gamma)^2 in Eq. (65), including the approximation-error terms. We also provide an explicit projection argument establishing ∣ξ′−ξ⋆(wk−1)∣A2≤Eapprox2\\|\xi'-\xi^\star(w_{k-1})\\|_A^2\le E_{\mathrm{approx}}^2. The resulting horizon dependence is now displayed explicitly in Theorems 1 and 2 and the corresponding intermediate bounds. As the reviewer noted, Eq. (70) already has this horizon dependence, so these corrections do not change the order of the actor-critic convergence bound.

W3: Proof typos.

We have corrected wK−1w_{K-1} to wk−1w_{k-1} in Eq. (61), replaced “Assumption 2” by “Condition 2 of Theorem 3” in Lemma 2 and the bound on (T3)+(T4), and replaced w0w_0 by the uniform weight vector uu in Appendix H.1 (formerly G.1).

W4 & Q1: PAVF comparison and the role of policy representation.

We have added the requested PAVF-style raw-parameter variant, RawParam, in Sec. 5.2 (Fig. 3). It replaces only our policy encoder with the full actor parameter vector followed by a learned projection, keeping the actor architecture and training recipe unchanged. Across Humanoid Stand, Walk, and Run (four seeds per task), RawParam retains much of our method’s performance, while Ours achieves higher mean returns at the common training endpoint on all three tasks.

Crucially, our approach differs from PAVF in more than its encoder. Although both use policy-conditioned critics and two-path actor gradients, a key distinction is which policies the critic learns to evaluate and provide gradients for. Ours and RawParam pair replay transitions with current actors, using the paired current policy for both critic conditioning and TD bootstrapping; historical-policy PAVF instead uses each transition’s collection-time policy. This current-policy training design is motivated by our off-policy evaluation and actor–critic analysis.

We additionally compare Ours and RawParam with a modernized historical-policy PAVF implementation in Fig. 5. On Humanoid Walk, PAVF shows little learning over the observed interval (approximately 63% of the planned budget), whereas both Ours and RawParam achieve substantially higher returns at the same environment-step budget. Taken together, these results support the practical value of our theory-inspired training design beyond the choice of encoder. See also “Ours vs PAVF” in our overall response (2/2).

W4 & Q2: Wall-clock cost and critic-side computation.

We agree that wall-clock cost is important when assessing sample-efficiency gains. Because these runs used heterogeneous GPUs and sharing conditions, we cannot provide a controlled comparison of wall-clock time per environment step from the recorded elapsed times.

To examine the contribution of critic-supervision volume, we added two controls in Fig. 4. Panels (a–c) compare Ours with SAC+ and TD3+ using tenfold larger critic minibatches. Panels (d–f) compare Ours-NoCross, which pairs each replay transition with one uniformly sampled current actor, with the standard-batch SAC+ and TD3+ baselines. Each comparison matches the nominal TD-example count per critic update. The learning curves retain advantages in environment-sample efficiency across the three Humanoid tasks, indicating that more TD examples per update alone do not explain the observed gains.

The theoretical concerns are addressed, and the analysis offers a novel unified treatment of off-policy actor–critic convergence. The new results in Figs. 3, 4, and 5 show a good comparison with the baselines.

However, the method uses around 10× more trainable parameters than SAC+/TD3+ (transformer's feed-forward width is not reported in Table 2, and I have taken it to be 256, please add this as well). Given this overhead, a controlled timing comparison (for example) wall-clock time per timestep or epoch on one GPU for the proposed method, SAC+ and TD3+ is necessary to weigh the sample-efficiency gains against cost and to make readers aware of the computational overhead.

My updated scores reflect the authors' efforts during the rebuttal, and I am open to increase it further, given the timing comparison.

AuthorsOpenReview ↗

Follow-up: computational cost and encoder configuration

Thank you for your thoughtful reassessment and for recognizing both that the theoretical concerns have been addressed and that the new experiments in Figs. 3–5 provide useful comparisons with the baselines. We agree that the sample-efficiency results should be accompanied by a controlled measurement of computational overhead.

We have added a single-GPU timing benchmark on Humanoid-Walk in Appendix J, Table 5. All four methods run sequentially on the same exclusive RTX 5090, with four parallel environments, batch size 256, and a common 10:1 critic-to-actor update schedule. Including environment collection, replay processing, and all parameter and target updates, Ours requires 70.38 ± 0.16 ms per environment transition, compared with 34.91 ± 0.16 ms for SAC+ and 32.09 ± 0.33 ms for TD3+, corresponding to 2.02× and 2.19× runtime. Ours-NoCross requires 69.27 ± 0.06 ms. We report mean ± sample standard deviation across three timing windows of 500 complete iterations after warm-up, with transitions counted across all four environments. The full protocol is provided in Appendix J.

We also corrected the inaccurate encoder-depth entry in Table 2 from four Transformer layers to one, and added the model width of 512 and feed-forward width of 2048. The profiled Ours/NoCross model contains 7,623,132 trainable parameters, compared with 992,565 for SAC+ and 987,167 for TD3+—approximately 7.68× and 7.72×, respectively. These counts include the full actor/critic ensembles and policy encoder, and exclude target-network replicas.

We hope these additional measurements and clarifications address your remaining concerns about computational cost and help inform your final assessment.

Comments on the paper

4 comments
~Haque_M_Ishfaq1OpenReview ↗

Comparison with Langevin Soft Actor Critic (LSAC)

Hi authors,

We had a highly related work from ICLR 2025 : Ishfaq, H., Wang, G., Islam, S., & Precup, D. (2025, May). Langevin soft actor-critic: Efficient exploration through uncertainty-driven critic learning. In International Conference on Learning Representations.

In this paper, we consider Langevin Monte Carlo based Thompson Sampling for uncertainty estimation in critic in SAC and DSAC for sample efficient RL with improved exploration. Many of the motivations in your paper aligns with ours. So, we would appreciate if you could consider citing our work as well as use LSAC as a baseline for your experiments.

We open sourced our code as well: https://github.com/hmishfaq/LSAC

AuthorsOpenReview ↗

Overall response to the Area Chair and all Reviewers (1/2)

We thank the reviewers for recognizing the potential significance of our theoretical results. To our knowledge, this is the first work to address four longstanding theory–practice gaps in off-policy actor–critic—target-based TD, partial feature coverage, evolving behavior policies, and replay sampling—within a common theoretical framework under linear function approximation. We establish actor–critic convergence with the first three ingredients and separately analyze replay sampling under i.i.d. assumptions. Even target-based off-policy actor–critic convergence is a nontrivial advance: Barakat et al. (2022) establish the on-policy result and explicitly leave the off-policy extension as future work. Our analysis addresses that problem and goes further, providing new guarantees for partial-coverage evaluation, policy-gradient generalization, and replay design.

The theory also guides a practical algorithm: we derive design principles in a tractable linear setting and test their usefulness with neural function approximation. A key departure from historical-policy PAVF (new Sec. 5.2) is which policies receive TD supervision: PAVF ties each transition to its collection-time policy, whereas Ours and RawParam evaluate current target policies using replayed transitions. RawParam changes only our encoder, preserving this training design; it is an ablation of our method, not a reproduction of PAVF. Its strong performance (new Figs. 3 and 5), alongside the weak learning of the historical-policy PAVF implementation at the common interim Humanoid Walk budget (new Fig. 5), supports a practical contribution beyond the choice of policy encoder. Our neural implementation retains strong performance in comparisons with SAC+/TD3+ matched for nominal TD-example counts per critic update (new Fig. 4), while omitting clipped double-Q targets, target-policy smoothing, and entropy regularization, demonstrating how linear theory can guide useful neural algorithm design.

Benchmark and baseline strength. We deliberately evaluate all eight DMC Dog and Humanoid tasks, requiring coordination of 38 and 21 continuous action dimensions, respectively, compared with 3–8 in common Hopper, HalfCheetah, Walker2d, and Ant benchmarks and 17 in Gymnasium/MuJoCo Humanoid (Appendix I, Table 4). Our comparisons use independently tuned SAC+ and TD3+ baselines that substantially improve on their vanilla counterparts on most tasks, as highlighted by the separate eight-task comparisons in Figs. 10–11. Our reported gains are thus measured against strong baselines on challenging, high-dimensional control tasks.


TL;DR — Revision highlights: We correct the proof constants and typos identified by MEEq and make spectral, mixing-time, and horizon dependencies explicit. New Sec. 5.2 shows that strong performance persists with PAVF-style raw-parameter encoding, whereas historical PAVF shows little learning in the partial Humanoid Walk comparison. Ours and Ours-NoCross also retain environment-sample-efficiency advantages over SAC+/TD3+ under controls matching nominal TD-example counts per critic update. New Appendix J reports model sizes and controlled single-GPU wall-clock costs alongside the sample-efficiency results. Major revisions are highlighted in blue in the revised manuscript.

Implementation correction. After submission, we identified and corrected an error in the implementation of our approach that double-counted an indirect contribution to the actor’s policy-input gradient. The correction modestly improves performance across all eight benchmark tasks. The revised manuscript reports the corrected results throughout, and all ablation variants and the PAVF comparison use the corrected implementation.

Scope of additional experiments. The additional experiments in Sec. 5.2 focus on the Humanoid tasks given the rebuttal’s time and resource constraints. We plan to extend these comparisons to the Dog tasks after the rebuttal and include the results in the final paper.

AuthorsOpenReview ↗

Overall response to the Area Chair and all Reviewers — Additional experimental results (2/2)

Encoding ablations. Fig. 3 shows that RawParam, which replaces our policy encoder with the full actor parameter vector and a learned projection, retains much of Ours’ performance. ActionFingerprint exhibits a larger performance drop on all three Humanoid tasks. Ours achieves the highest mean return among these three variants at the common training endpoint on each task. These results show that our training design remains effective with PAVF-style raw-parameter inputs, while our richer policy representation provides an additional empirical benefit.

TD-supervision controls. We examine the role of TD-supervision volume through two complementary controls: increasing SAC+/TD3+ batch sizes by 10× to match Ours’ nominal TD-example count per critic update (Fig. 4a–c), and reducing Ours to one current-policy pairing per replay transition (Ours-NoCross) to match the standard baselines on the same measure (Fig. 4d–f). The learning curves retain sample-efficiency advantages in both comparisons across the three Humanoid tasks. These results support an advantage beyond simply using more TD examples per update.

Computational cost update. We have added a controlled single-GPU benchmark on Humanoid Walk (Appendix J, Table 5). On the same exclusive RTX 5090, Ours requires 70.38 ms per aggregate environment transition, versus 34.91 ms for SAC+ and 32.09 ms for TD3+, corresponding to 2.02×/2.19× runtime; Ours-NoCross requires 69.27 ms. Timing includes environment collection, replay processing, and all parameter and target updates. Ours has 7.623 million trainable parameters, approximately 7.7× the baselines. These timings use a common 10:1 critic-to-actor update schedule and therefore a 11:1 UTD. Appendix J provides the full protocol, model sizes, and peak GPU memory, and Table 2 now reports the corrected encoder configuration.

Ours vs PAVF. PAVF already introduced policy-conditioned value functions and two-path actor gradients. Our theoretical evaluator learns QπQ^{\pi} for a designated target/reference policy π\pi from data collected by a different behavior policy μ\mu; the actor–critic analysis then controls gradient error as the actor evolves away from that reference. This separates the policy being evaluated from the source of the replay data. In Ours and RawParam, each replay transition is paired with the current actor ensemble: for each pair, the critic is conditioned on the current policy and the TD bootstrap uses that same policy. Historical-policy PAVF instead uses the transition's collection-time policy for both conditioning and bootstrapping. The practical change is therefore which policies the critic learns to evaluate and provide gradients for, not merely how policies are encoded. This training choice was motivated by our analysis, rather than an empirical search for modifications to PAVF.

Our modernized historical-policy PAVF comparison retains collection-time conditioning and historical-policy bootstrapping, using a shared learned projection to accommodate the 89,621-parameter actors. It still requires storing and reconstructing historical policies. Figure 5 reports partial historical-PAVF results on Humanoid Walk, covering 83.25% of the planned interaction budget. At the latest evaluation point shared by all four seeds (124,875 steps per parallel environment), PAVF achieves a mean return of 2.4, compared with 502.3 for Ours and 442.9 for RawParam at the same interaction budget. Historical PAVF shows little learning over the observed interval, while both current-policy variants achieve substantially higher returns.

AuthorsOpenReview ↗

Relation to LSAC and scope of comparison

Thank you for bringing LSAC to our attention. We agree that it is relevant to our discussion of uncertainty-driven exploration and will cite it there. Our main theoretical contribution is a convergence framework for off-policy actor–critic under linear function approximation, which motivates the training design of our policy-conditioned functional critics. LSAC focuses on Langevin-based critic learning for exploration and does not address these convergence questions or use an explicitly policy-conditioned critic. We appreciate the baseline suggestion, but cannot complete a controlled LSAC comparison within the remaining rebuttal period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.