acceptodds
Under review as a conference paper at ICLR 2027

Squared Family MDPS: Provably Efficient RL with Tractable Representations

Abstract

Regret bounds in continuous reinforcement learning require structural assumptions on the underlying MDPs. Many model-based parametric approaches are either difficult to theoretically analyse or hard to implement. We address this issue by proposing the Squared Family MDPs of which the transition kernels are parameterised by a family of tractable probabilistic models called squared families. Theoretically, we derive a regret bound, sub-linear in total time , linear in feature dimension and horizon length . Practically, we propose Representation RL with the Squared Neural Family (RESNEF), which uses Maximum Likelihood Estimation to learn the transition density and extract useful representations for downstream RL tasks. RESNEF brings up to 40% performance improvements compared with state-of-the-art baselines across diverse continuous control tasks.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Regret Bounds, Continuous Control

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RPPgOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper models transition densities with normalized squared feature expressions. It proposes Squared Family UCRL, an optimistic RL algorithm with a sublinear regret bound under known-feature, realizability, and regularity assumptions. Its practical method, RESNEF, uses squared neural families with analytically computable normalizers to learn transition models by maximum likelihood. It also expresses this neural construction as a linear transition model through a quadratic feature expansion. Experiments compare RESNEF and a random-feature variant with representation-learning baselines on continuous-control tasks. The regret guarantee is limited to Squared Family UCRL and does not extend to RESNEF.

Critical strengths
  1. For the selected neural features and base measures, the squared family gives nonnegative transition densities whose normalizers can be computed analytically. This allows direct maximum likelihood training and addresses the normalization difficulty in expressive energy-based transition models.

  2. The authors construct likelihood-based confidence sets using Hellinger geometry and a martingale argument. They use these sets to establish sublinear regret for an optimistic RL algorithm under the stated assumptions.

Critical weaknesses
  1. There is a gap between the regret analysis and the practical algorithm. The theoretical algorithm is not directly implementable as presented. RESNEF uses neural networks and relaxes the theoretical assumptions, so the regret guarantee does not cover its design.

  2. The empirical gains are inconsistent across tasks. Results from four random seeds also provide limited evidence about how reliably the method improves performance.

Questions for authors

I understand that RESNEF does not satisfy the assumptions used to establish the regret bound. Could you evaluate RESNEF on a MDP with a computable optimal policy, measure its cumulative regret as the number of environment steps increases and compare the observed regret with the regret bound in Theorem 4.6?

5 comments

To reviewer RPPg,

Thank you for the review and for highlighting the theory-to-practice gap. We agree that the regret theorem does not directly cover RESNEF, and we already identify this as a limitation in Section 7. However, we would like to clarify why we view the theoretical and practical components as complementary rather than disconnected. It is actually our intention to close the theory to real gap along this line of research, as existing empirical works needs to bypass the MLE computation suggested by the theory, but we are the first to propose a theoretical framework that retains sharp regret bound under MLE and keeps MLE in the practical algorithm. Please see each of our point below.

W1: Gap between Squared Family UCRL and RESNEF

The theoretical contribution asks whether squared-family transition models admit statistically efficient RL under a standard realizable, known-feature setting. Squared Family UCRL answers this question by combining MLE concentration, confidence sets, optimistic planning, and an elliptical-potential argument.

The practical contribution addresses a different obstacle: making the likelihood-based representation learning mechanism itself tractable. Existing practical representation-learning methods such as CTRL and DiffSR are motivated by related low-rank/linear-MDP theory, but replace MLE with NCE or score matching because their probabilistic normalisers are intractable. SNEFY provides analytically tractable normalisation, allowing RESNEF to retain the MLE mechanism suggested by the theory while learning nonlinear representations.

We therefore do not claim that Theorem 4.6 is a regret guarantee for RESNEF. Rather, the theory establishes squared families as a statistically well-behaved transition class under MLE, and RESNEF demonstrates that the same tractability property can be exploited for learning representation using MLE and achieve good performances especially when compared to other representational approaches. We will sharpen this distinction in the introduction and limitations section.

Q1: Evaluating RESNEF against Theorem 4.6

We appreciate the suggestion to measure cumulative regret on an MDP with a computable optimal policy. Such an experiment could be useful for studying the empirical behaviour of the theoretical algorithm. However, directly comparing RESNEF’s regret to the bound in Theorem 4.6 would not constitute a meaningful verification of that theorem, because RESNEF neither implements the confidence-set optimistic planner analysed in Theorem 4.6 nor satisfies all of its assumptions. Agreement with, or violation of, the bound by RESNEF would therefore say little about the correctness or tightness of Theorem 4.6.

A faithful empirical test of the theorem would instead require implementing Squared Family UCRL on a deliberately constructed MDP with known features and realizable squared-family transitions, and repeatedly solving the confidence-set-constrained optimistic planning problem. That computational difficulty is precisely one of the motivations for introducing RESNEF as a practical relaxation. We will make this distinction explicit so that the theoretical guarantee is not interpreted as applying to the practical algorithm.

W2: Empirical variability and number of seeds

We agree that the gains are not uniform across tasks, and we will moderate the corresponding language in the abstract. Nevertheless, the overall empirical evidence is stronger than the weakest individual cases suggest.

Against the closely related representation-learning baselines (i.e. DiffSR and CTRL), RESNEF/RESNEF-RF achieve the best performance on 6 of 8 Gymnasium tasks and 4 of 5 DMC tasks. The MLE-versus-score-matching ablation further isolates the estimation objective: replacing MLE with score matching while keeping the RESNEF architecture otherwise fixed reduces performance. These results support our narrower claim that tractable MLE is a useful mechanism for learning transition representations, not that our method uniformly dominates every model-free or representation-learning algorithm. Meanwhile, our results are also not dominated by SAC either, as many settings our approaches perform better than SAC, hinting the usefulness of learning good representations.

We use four independent random seeds for all methods and report standard errors. Four seeds are also used by the compared baselines and are common in continuous-control evaluations under substantial computational cost.

Summary

Thank you again for the review. We hope the clarification above makes the big picture more clear: the theorem establishes sample-efficient learnability of the squared-family model class under explicit assumptions, while RESNEF exploits the same tractable likelihood structure to move practical representation learning closer to that theoretical mechanism. If these clarifications address the concerns, we would appreciate reconsideration of the rating as appropriate.

Thank you for the clarification. I understand that the paper does not claim to fully close the theory–practice gap. However, the connection between the theoretical and empirical contributions remains insufficiently established, and I remain unconvinced that they jointly address a central research question.

My request concerned measuring RESNEF’s empirical regret. I acknowledge that Theorem 4.6 does not apply to RESNEF. Nevertheless, this evaluation would provide useful evidence about its learning behavior and help assess the practical relevance of the theoretical framework.

I appreciate the authors’ response, but my concerns remain unresolved, and I therefore retain my original rating.

Thank you for you reply. We respectfully disagree that the theoretical and empirical contributions lack a sufficiently strong connection or fail to address a central research question. A central objective of our work is to establish squared families as a theoretically sound class of transition models for sample-efficient RL, while demonstrating that their tractable likelihoods enable practical representation learning that remains closely aligned with the theoretical formulation. Specifically,

  1. Shared probabilistic framework. Definition 5.1 builds on top of SNEFY, a subclass of squared families defined in Definition 4.1. We have derived definition 5.1 from the feature embedding setting given just below Definition 4.1. We have also shown just below Definition 5.1 that it can be re-written in pretty much the same format as the one in Definition 4.1.
  2. Preservation of MLE. Definition 5.1 provides tractable normalisation term, which allows us to perform parameter estimations using MLE, essentially line 3 of the theoretical Squared Family UCRL algorithm. Retaining MLE for the same density models from theory and practice is therefore a substantive step towards bridging the theory–practice gap, rather than merely a superficial connection between two separate contributions.
  3. Connection to the random feature setting. Our theory considers known feature setting, and in fact, the variant RESNEF-RF freezes the kernel after random initialisation (acting like the random feature setting provided after Definition 4.1), making Vθ1(s,a)V_{\theta_1}(s,a) (equivalent to the parameter θ\theta in Definition 4.1) the only learnable part. This makes RESNEF even closer to the theoretical formulation and in fact, it also performs well across many tasks as shown in the experiment while saving half of the training time budget compared to RESNEF (see our replies to 6taw).

There is always some sort of gap between sharp theory and scalable algorithm. In our case, the biggest gap comes from the planning module, as RESNEF replaces the planning around the confidence set (line 4 and 5 of Squared Family UCRL) with the actor critic framework. Acknowledging such gap exists, we hence provide empirical evidence by comparing RESENF against two strongest representational actor critic methods CTRL and DiffSR and one classic model free actor critic algorithm SAC. The fast-growing evaluation returns of RESNEF and RESNEF-RF hence demonstrates their sample efficiency.

Constructing a meaningful continuous-control benchmark with a computable optimal policy is challenging, and such evaluations are not generally required in this line of representation-learning research (Jin et.al., 2020, Agarwal et.al., 2020, Chowdhury et al., 2021, Uehara et. al., 2021, Zhang et.al., 2022, Shribak et al., 2024). Even if the regret is known, as we have mentioned in the last reply, comparing RESNEF’s regret against Theorem 4.6 is not informative because the theoretical algorithm does not match the practical algorithm. Since small cumulative regret corresponds to fast growing returns, the most widely adopted method to test the sample efficiency of empirical algorithms is to compare their return-to-environment-interaction curves against other baselines. This is exactly what we have done in our experiments to show the effectiveness of sticking to squared family MDP and estimating parameters with MLE for solving RL problems. Better yet, our empirical evidence is not only from toy simple problems, but more realistic and scalable continuous control problems across wide range of settings.

In summary, we respectfully maintain that our theoretical and empirical contributions address a coherent research question from complementary perspectives: whether squared-family transition models enable statistically efficient RL in theory, and whether their tractable likelihood structure can be exploited to improve practical representation learning. We have explicitly identified the remaining theoretical limitations and do not claim a regret guarantee for RESNEF. We believe the concrete methodological connections and extensive empirical evidence establish a considerably stronger theory–practice relationship than the reviewer's latest assessment suggests.

Reviewer_oELBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes modeling transition dynamics using "squared families"—probability densities constructed by squaring a linear combination of features and subsequently normalizing the result. For a constrained, feasible setting with known features, the authors introduce "Squared Family UCRL" and derive a regret bound using maximum likelihood estimation, a likelihood-ratio martingale, Hellinger-distance confidence sets, and elliptical potential arguments. The practical method, RESNEF, learns conditional squared neural density models and incorporates their representations into an actor-critic algorithm. A variant utilizing random features, RESNEF-RF, freezes the next-state feature map to reduce normalization costs. The experiments cover eight Gym-MuJoCo tasks and five DeepMind Control tasks, comparing the results against SAC, CTRL, and DIFFSR across four random seeds. Additional experiments investigate maximum likelihood estimation versus score matching, the impact of feature dimensions, and computational overhead. The practical method, RESNEF, learns conditional squared neural density models and employs their representations within an actor-critic algorithm. While the approach is promising, the manuscript contains several mathematical inconsistencies and ambiguities regarding implementation that must be addressed before its primary claims can be evaluated with confidence.

Critical strengths
  1. Through a coherent normalization process, probability-based representation learning becomes possible without having to replace the probability itself with a surrogate objective. The quadratic construction also ensures non-negative density estimates.

  2. The likelihood-ratio martingale in Appendix C.3 addresses adaptively collected observations, and the quadratic formulation of the Hellinger distance in Lemma C.11 establishes a natural link between estimation error and parameter geometry. These are valuable components, even though some subsequent statements may require correction.

  3. Testing learned versus frozen next-state features, comparing the likelihood-based approach with score matching, and varying the representation dimensions all help in evaluating the proposed design. It is also beneficial to report results across four random seeds, including standard errors.

Critical weaknesses
  1. If the other problem constants are considered fixed, the bound provided in Theorem 4.6 yields: O~ ⁣(dHT+H3/2dT)=O~ ⁣(HTmax⁡{d,dH}).\widetilde O\!\left(dH\sqrt T+H^{3/2}\sqrt{dT}\right)=\widetilde O\!\left(H\sqrt T\max\{d,\sqrt{dH}\}\right). This dependency is explicitly addressed in Appendix C.5 and Appendix D. Consequently, the simplified rate O~(dHT)\widetilde O(dH\sqrt T) presupposes a scenario such as d≥Hd\ge H; this qualification is missing from the abstract and the conclusion. The statements within the proofs also require revision. Lemma C.14 omits the factor HH present in Lemma C.13 and asserts that the conclusion holds for arbitrary sequences of parameters. Such a claim cannot be correct in general: repeatedly selecting a fixed, incorrect model can result in a Hellinger error that grows linearly with TT. The lemma necessitates the conditions on the confidence sets that are also employed in its proof. Furthermore, Theorem 4.5 claims simultaneous coverage for all KK, whereas Appendix C derives a result for a fixed time horizon using a fixed covering net, with α=1/T\alpha=1/T subsequently chosen. The stated confidence radius and the assertion regarding an infinite time horizon require an additional argument or a more precise, restricted formulation. While these points might be resolvable, they concern fundamental guarantees.

  2. The theoretical parameter set requires zθ(s,a)=1z_\theta(s,a)=1 for every (s,a).(s,a). Proposition C.1, by contrast, fixes a state-action pair and scales θ\theta by 1/zθ(s,a)1/\sqrt{z_\theta(s,a)}. While this yields a pointwise normalization result, the scaling generally depends on (s,a)(s,a). Consequently, it does not establish the existence of a single, common parameter that satisfies all normalization conditions simultaneously. As a result, the proposition does not support the underlying assumption that restricting the model to the global set Ω\Omega preserves general conditional expressivity. Nor does it substantiate an approximation in which both the parameter bounds and Assumption 4.4 are maintained. Fixed smoothing introduces an additional constraint: every model must satisfy Pθ(A∣s,a)≥ϵμ(A)P_\theta(A\mid s,a)\ge\epsilon\mu(A). Thus, for a fixed ϵ>0\epsilon>0, it is impossible to approximate—with arbitrary precision—target distributions that assign zero probability to a set with a positive base measure. When discussing approximation, a distinction should therefore be made between smoothed target distributions and unrestricted transition kernels.

  3. Equation (13) contains the term wj⊤wjw_j^\top w_j, which is multiplied by the sum of two sine terms. In its current form, this expression is generally not symmetric with respect to swapping II and jj, whereas $ K_{ij}=\mathbb E_\mu[\xi_i(X)\xi_j(X)] $ must be symmetric. The factor should instead be wi⊤wjw_i^\top w_j. Direct numerical integration confirms this correction, which is also consistent with the final derivation presented in the cited SNEFY paper. Furthermore, Eq. (13) refers to the shifted activation function Snakea−1/(2a)\mathrm{Snake}_a - 1/(2a), while the description of the experiments specifies Snakea\mathrm{Snake}_a. These activation functions result in different kernels. Since precise normalization is of central importance to the method, the authors should clarify which activation function and which formula were actually implemented, and whether the reported likelihood values ​​are correctly normalized.

  4. Equation (12) evaluates the target critic at the point shks_h^k, whereas the Bellman target should instead utilize the observed subsequent state sh+1ks_{h+1}^k. If this is a typo, it should be corrected, and the implementation should be verified accordingly. Furthermore, the relationship between the two critics, their respective target networks, and the minimum operation employed in Algorithm 2 requires a precise description. In Proposition 5.2, μ(ds′)\mu(ds') is contained within νθ2(s′)\nu_{\theta_2}(s'); however, in Equation (11), this expression is integrated once more with respect to μ(ds′)\mu(ds'). The paper should adopt a consistent approach here—either by using a vector-valued density (integrated with respect to μ\mu) or by employing a directly integrated vector-valued measure. Finally, ϕθ1\phi_{\theta_1} depends on θ2\theta_2 via its normalization factor, which renders the subscript θ1\theta_1 misleading. These distinctions are essential for understanding both the likelihood calculation and the gradient updates.

Questions for authors
  1. Could you provide corrected formulations of Theorems 4.5–4.6 and Lemma C.14, in which the dependence on the time horizon, the choice of the covering net, the confidence events, and the conditions on the parameter sequence are explicitly stated?

  2. Could you provide an example of a non-trivial conditional model family that satisfies the global normalization condition and Assumption 4.4, and explain the approximation guarantee that applies to this family?

  3. Does the implementation use the corrected term wi⊤wjw_i^\top w_j from Equation (13)? Is the activation function "Snake" or "shifted Snake"? Furthermore, have the normalization factors and their gradients been numerically verified?

  4. Does the implemented critic target use the variable sh+1s_{h+1}? Please specify the complete critic target, the handling of terminal states, and the roles of the two critics and the target networks.

  5. What specific result is intended for the mixture model in the case where m=1m=1, and how are the remaining cross-terms—as well as potentially negative coefficients—accounted for?

  6. Could you include comparisons with similar architectures and additional auxiliary losses, and investigate the significant performance degradation observed in the "Humanoid" and "Hopper-Stand" environments?

  7. How are the base measure of the Gaussian mixture model and the smoothing parameter initialized and updated? Does the critic use YY, vec⁡(Y)\operatorname{vec}(Y), or the normalized, high-dimensional feature (lifted feature) ϕ\phi as input?

3 comments

To Reviewer oELB

We sincerely thank the reviewer for the detailed and technically careful feedback. Several of the identified issues are genuine presentation or statement-level errors, and they can be easily fixed without modifying the central intended results or contributions in the paper.

W1 & Q1

  1. We agree with the reviewer to report the whole regret rate presented in the Appendix as, $ \widetilde{O}!\left(H\sqrt{T}\max(d,\sqrt{dH})\right). $ The simplified O~(dHT)\widetilde{O}(dH\sqrt{T}) rate holds when d≥Hd\ge H, which is previously thought to be the common case in RL, as the number of parameters is typically larger than the horizon size.

  2. For lemma C.13-14, we emphasize that these are typo errors in the intermediate lemma statement, not in the final regret substitution. The final derivation also retains the required HH dependence, please see the last inequality when we applied Lemma C.14 in Appendix C.5. Theorem 4.6 proof is conditioned on the good events, we just need to restate Lemma C.13 and C.14 to condition on the good events to avoid the linear growth of Hellinger error problem.

  3. Yes, we should state theorem 4.5 as a finite-horizon simultaneous guarantee: for any prescribed KK (equivalently T=KHT=KH), with probability at least 1−δ1-\delta, the true parameter belongs to all confidence sets generated up to episode KK. We will remove the infinite intersection of the confidence set appeared in the end of Theorem 4.5. This is also the guarantee required by Theorem 4.6. Since everything is now finite, we are free to pick the alpha-net precision scalar α\alpha to match what the proof needs.

W2 & Q2

We agree that Proposition C.1, as currently written, establishes a pointwise statement and should NOT be interpreted as proving that a single shared parameter can simultaneously satisfy zθ(s,a)=1z_\theta(s,a)=1 for every (s,a)(s,a) while preserving unrestricted conditional universal approximation. We actually do not need normalised parameter set for Proposition C.1 to hold for the proof to go through. Please read our revised Propocision C.1 in the rebuttal revision of the paper and the remark followed.

Furthermore our theoretical analysis is under the realisable setting: the target transition kernel is assumed to belong to the same smoothed model class in Definition 4.1, as this is a standard practice.

Importantly, Proposition C.1 is not used to prove the regret bound or any of the subsequent result. Assumption 4.4 is a separate regularity condition and Appendix D provides a non-trivial sufficient condition for it.

W3 & Q3

Indeed, Equation (13) contains a typo: the asymmetric term should involve wi⊤wjw_i^\top w_j. Our experiments use the standard Snakea\mathrm{Snake}_a activation, not only the shifted Snakea−1/(2a)\mathrm{Snake}_a-1/(2a) form displayed around Eq. (13). We will add the explicit kernel expression used in the implementation and clearly distinguish the two forms.

W4 & Q4

Yes, Eq. (12) contains a typographical error: the target critic must be evaluated at the observed next state sh+1s_{h+1}, not shs_h. RESNEF uses two critics and corresponding target critics. The target networks are updated by Polyak averaging, and the minimum of the two target Q-values is used in the Bellman target to mitigate overestimation; the actor is likewise trained against the minimum of the two current critics.

Q5

When m=1m=1, the quadratic feature has rank one, reducing the expressive capacity of the matrix-valued squared model and we observe subtantial drop and hence not included it in Figure 7.

Q6

We chose CTRL and DiffSR because they are the closest practical representation-learning baselines built around linear-MDP-style transition representations. We are not aware of an existing architecture that provides a closer like-for-like likelihood-based baseline.

In Humanoid and Hopper-Stand, RESNEF-RF substantially outperforms RESNEF. A plausible explanation is optimisation coupling: in RESNEF, both the state-action feature and next-state feature are learned simultaneously, so the kernel and normaliser change throughout training. This creates a moving optimisation landscape for the likelihood objective, induce additional complexities.

Q7

For GMM the means are drawn from standard Gaussians, log scale is set to 0 and mixing weights is set to 1/8. The practical smoothing quantity λ\lambda is initialised to 1 and gets squared after to retain non-negativity. For the critic input, the implementation uses the flattened quadratic state-action representation derived from YY rather than explicitly constructing the full normalised high-dimensional lifted feature.

Summary

We again thank the reviewer for the detailed technical review. We believe these points helped us to make the paper more rigorous without changing the original claim. We hope these revisions resolve the soundness concerns, and we would appreciate reconsideration of the rating if the reviewer finds the corrected statements satisfactory.

I think the authors for their responses but prefer to keep my current score.

Thank you for your response. We sincerely appreciate the detailed and constructive feedback in your initial review, which has helped us improving the clarity and rigour of the paper. We take that we have addressed each of your seven questions and the associated weaknesses, including corrections to the theoretical statements, clarification of the implementation, and additional explanations of the empirical results. These changes will be made to our revised paper and shouldn't be any substantial modifications of the content. If any of your original concerns remain unresolved, we would greatly appreciate if you can identify the specific points and explain why our responses are insufficient. If you have concerns on our practical performance or the theory to real gap, please see our responses to Reviewer 6taw and Reviewer RPPg, which they seem to satisfy. Otherwise we respectfully ask that you reconsider whether the original rating remains justified in light of the corrections and clarifications provided. This would also be helpful for the AC to understand the reasons behind the ratings.

Reviewer_6tawOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces smoothed squared-family transition models and derives a regret bound for an optimistic model-based algorithm under known features, realizability, and uniform geometry assumptions. It separately proposes RESNEF, which learns squared-neural density representations for an actor-critic agent, and RESNEF-RF, which freezes the next-state features. The results show gains on several tasks but substantial task dependence; the regret theorem does not cover the practical algorithms.

Critical strengths
  1. Expressive dynamics with tractable likelihoods. The squared construction ensures non-negative transition densities and, for the chosen SNEFY parameterizations, analytically computable normalization. This enables direct maximum-likelihood training, addressing the intractable partition functions that limit some exponential-family approaches to continuous RL.

  2. A direct bridge from density modeling to value learning. Proposition 5.2 derives transition features algebraically from the squared model, giving likelihood-based representation learning a concrete connection to value-function structure, making the adaptation of squared families to RL consequential.

Critical weaknesses
  1. The practical efficiency is not sufficiently established. The primary comparisons fix environment interactions, not elapsed training time. Figure 3 reports 0.681 seconds for RESNEF versus 0.051 for SAC and 0.120 for DiffSR per ten training iterations on HalfCheetah: approximately 13.4× and 5.7× the measured cost. Without end-to-end return-versus-time curves, it remains unclear whether the proposed algorithm’s performance gains persist under an equal training-time budget.
  2. The algorithm seems to lack robustness, as the performance looks very task-dependent. RESNEF obtains 533 versus SAC's 2,434 on Humanoid, and 303 versus CTRL's 898 on Hopper-Stand. The claimed gains of up to 40% do not fully reflect these substantial performance drops on certain tasks.
Questions for authors
  1. What does Figure 3's timing include, and what hardware and update-to-data ratios were used? Additional timed runs would strengthen the practical-efficiency claim if it can establish return versus end-to-end training time comparisons for the baselines and proposed methods.
  2. How much of RESNEF’s improvement comes from likelihood-based transition learning, rather than the critic’s training signal? Since the representation is trained jointly by MLE and the critic’s TD loss, can the authors clarify their respective contributions? A comparison with critic gradients blocked from the transition representation would help establish whether the gains support the proposed density-learning mechanism or primarily reflect reward-driven representation learning.
3 comments

To Reviewer 6taw,

Thank you for the careful review. We appreciate that the reviewer clearly identified the central motivation and strengths of our work, particularly the tractability of likelihood-based learning and the connection from density modelling to value learning. We address the concerns below and will revise the paper accordingly.

W1 & Q1: Practical efficiency and wall-clock cost

Following the reviewer’s suggestion, we measured the full end-to-end wall-clock training time for all methods over 200K environment steps on HalfCheetah using an NVIDIA B200:

MethodTraining Time
SAC26m 31s
DiffSR39m 55s
CTRL50m 41s
RESNEF-RF1h 07m 38s
RESNEF2h 05m 20s

We agree that these results is more approporiate than the short runs we did in the paper. Further, we want to stress that our primary empirical claim concerns sample efficiency: the return achieved for a fixed number of environment interactions. For this purpose, return-versus-environment-step curves are the standard and most direct comparison, because all methods receive the same amount of interaction data. This approach is widely adopted by most RL settings, including all the baselines we compare here. In our case, the trade off of being sample efficient is the extra computational cost since MLE relies on computing squared family normalisation terms in closed forms. Importantly, the wall clock time shows that RESNEF-RF saves about half of training time compared to RESNEF while retaining its sample efficiency as shown in the experiment results, hence it is made very practical.

We would like to also clarify that the additional SNEFY kernel and normalisation computations are required only during training. At evaluation time, only the learned actor is executed, so RESNEF and its variants introduce no additional inference-time kernel or density-normalisation cost.

W2: Task-dependent performance

We want to emphasis that the experimental comparison is especially informative against other representation-learning baselines. Across Gymnasium, RESNEF/RESNEF-RF outperform the other representation-learning methods (excluding SAC) on 6 of 8 tasks, and across DMC on 4 of 5 tasks.

Our claim is therefore not that RESNEF uniformly dominates all RL algorithms, but that tractable likelihood-based transition representation learning is competitive and often substantially stronger than closely related representation-learning approaches. This interpretation is also supported by our MLE-versus-score-matching ablation: keeping the architecture fixed while replacing MLE with score matching degrades performance. We will revise the abstract and discussion to make this scope precise.

Q2: Contribution of MLE versus critic gradients

We performed the requested ablation by blocking critic gradients from flowing into the transition representation while leaving the MLE objective unchanged. The training plots can be viewed in the rebuttal revision of the paper, Figure 8.

Without critic signal, RESNEF achieves 6219 ±\pm 984 in half cheetah, 3293 ±\pm 842 in Walker and 1428 ±\pm 498 for Hopper. In walker and half cheetah, we observe the training curve being consistently a bit lower than the standard RESNEF. In Hopper, RESNEF without critic signal achieves very high reward early on but the performance starts to fluctuate later, hinting the need to constraint the squared family density learning to avoid the scale invariant problem. These results show that MLE alone is sufficient to learn a useful transition representation in the easier setting, while allowing the critic signal to shape the representation becomes increasingly important on harder tasks.

This behaviour is consistent with a structural property of squared families: because the density is invariant to certain parameter rescalings, multiple parameterisations can represent the same transition distribution. MLE enables density learning but does not constraint the representation stays the same throughout training. It is possible that a different representation suitable for the density is found, but if the critic never sees the representation before, it will behave poorly and induce fluctuations in the actor’s performance. The critic gradient helps select, among statistically equivalent or near-equivalent representations, one that is useful for the actor-critic objective. Thus, the ablation does not undermine the role of likelihood learning; rather, it shows that MLE supplies the probabilistic representation objective while the critic resolves downstream representation selection.

Summary

We thank the reviewer again for the constructive suggestions. We hope these additional measurements and ablations address the concerns regarding computational cost and the role of critic gradients. If the reviewer finds these clarifications satisfactory, we would appreciate reconsideration of the confidence/rating as appropriate.

Hi authors, thanks for putting in the time to run the new baselines and ablations.

The wall-clock timings on the B200 help. Seeing that RESNEF-RF cuts the training cost at least in half, and clarifying that inference time isn't impacted, resolves my concerns about practical efficiency. The trade-off for sample efficiency makes a lot more sense now. The critic gradient ablation is also a great addition, which shows the necessity of the critic to stabilize training and resolve scale-invariance.

You've thoroughly addressed my questions and improved the paper's empirical backing. Since my initial reviews already reflected a positive view of the core work, I will be keeping my current scores. Good luck with the rest of the review process!

We are glad we have addressed your concerns and thank you once again for your positive ratings and useful reviews.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.