acceptodds
Under review as a conference paper at ICLR 2027

SUNDER: A General Framework for Decoupled State–Policy Optimization in Offline Reinforcement Learning

Abstract

Distribution Correction Estimation (DICE) methods directly optimize policy-induced state-action occupancies, enabling distributional objectives beyond standard behavior-regularized reinforcement learning. However, joint occupancy regularization couples state visitation and action selection through a single regularization strength, even though these two forms of distribution shift may require different control. Naively separating them breaks the explicit affine structure underlying the standard DICE strong-duality argument. We introduce SUNDER (State–policy UNcoupling for Distribution Estimation via Ratio factorization), a general framework that separately controls state visitation and policy deviation. Although the resulting state–policy formulation is non-affine, we show that it admits a hidden convex structure that preserves strong duality and enables tractable elimination of the explicit optimization variables. We instantiate SUNDER for reward maximization, state-entropy maximization, and learning from observations. Under mismatches between state coverage and action quality, Opt-SUNDER outperforms OptiDICE across six maze domains, while achieving the best average performance among competitive recent baselines on 27 Minari tasks. Ent-SUNDER consistently outperforms SEMDICE across seven maze and four DMC domains with skewed dataset state distributions, and LfO-SUNDER achieves competitive performance on MuJoCo and Adroit tasks.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, occupancy measure optimization, state-policy decoupling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Z24WOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Classical DICE algorithms apply joint state visitation and action selction for constructing occupancy regularization. In this mannuscript, the authors propose SUNDER (State policy UNcoupling for Distribution Estimation via Ratio factorization) to seperately contril state visitation and action selection. The authors shows the appealing properties of SUNDER such as hidden convexity and strong duality preserving. Evaluated on Maze, Minari / D4RL benchmarks, SUNDER methods (Opt‑SUNDER, Ent‑SUNDER, LfO‑SUNDER) fully demonstrate the superiority.

Critical strengths
  • The studied problem is interesting and critical. State‑distribution shift and policy action deviation are two distinct dimensions, and the authors point out an important issue to study.

  • The theoretical contribution of the proposed method is appealing. The paper reveals hidden convexity via perspective‑convexity, preserving strong duality.

  • The manuscript is well-organized, and the proposed method is one unified framework for implementing various tasks: reward maximization, state‑entropy maximization, learning‑from‑observations.

Critical weaknesses
  • Compared to vanilla DICE, SUNDER optimizes dual objective for both state and policy regularizers, which leads to extra complexity. The paper does not report runtime or GPU memory usage, lacking quantitative computational‑cost comparison against OptiDICE and other baselines.

  • All theoretical hidden‑convexity and non‑recoverability theorems are hold at population level, assuming exact occupancy representation. However, we use neural‑network for implementation, which suffers function‑approximation error, sampling noise and empirical f‑divergence estimation bias. It is suggested to provide further analysis on practical error propagation and estimation.

  • The theoretical prerequisites are somehow strong and hard to verify in practice. For instance, hidden convexity and strong duality require proper, closed, convex f‑divergence generator. However, empirical sample‑estimated f‑divergence can violate conditions.

Questions for authors
  • The experiment studies are performed on Minari / D4RL simulation; dataset. Is there any real-scenario experiment?

  • The performance of LFO-SUNDER is the worst among all competitors under MuJoCo tasks, any explanations on this issue?

  • It seems the performance of SUNDER depends on policy extractor capacity, while Flow‑Matching extractor is much better than guassian. Any ablation on how to select policy extractor, especially in cases where policy class lacks sufficient capacity to fit target density ratios.

5 comments

We thank the reviewer for their insightful comments and have updated the manuscript accordingly.

[W1. Computational Cost of State–Policy Decoupling]

Thank you for pointing this out. We have added a computational-cost comparison to Appendix D.5 of the revised manuscript.

We compare against OptiDICE because it is the joint-regularization counterpart of Opt-SUNDER. The two methods use identical per-network sizes and the same policy-extraction procedure, so the comparison isolates the cost attributable to state–policy decoupling. The non-DICE baselines belong to a different algorithm family and are run from the official GFP implementation, so a wall-clock comparison with them would reflect those differences rather than the cost of decoupling. Both methods were trained on HalfCheetah on a single NVIDIA RTX PRO 6000 Blackwell GPU for 500K updates.

MethodTraining Time (s)Peak GPU Memory (MiB)
OptiDICE300.67754
Opt-SUNDER (Ours)437.77884
Overhead+45.6%+17.2%

The overhead comes from two additional state-only functions that Opt-SUNDER trains alongside VV: the policy-normalization multiplier λ(s)\lambda(s) and the auxiliary state function ed^(s)e_{\hat d}(s) (Appendix E.1). The optimization remains a single-level minimization with one extra regression per update; it adds no inner loop, adversarial step, or extra pass over the data. In absolute terms, the overhead is about 2.3 minutes of training time and 130 MiB of memory. The table measures the DICE training stage. Policy extraction and inference use the same weighted flow-matching procedure and policy architecture for both methods, so their cost is identical.

The additional coefficient also does not require a larger search. With an equal three-candidate budget and α\alpha fixed to 1, Opt-SUNDER still outperforms OptiDICE (83.96 ± 0.49 vs. 77.94 ± 0.73 over 27 tasks; Table 7), and choosing (α,β)(\alpha, \beta) from dataset statistics without per-task tuning retains most of the performance (84.01 vs. 85.44; Table 12).

[W2. Practical Error Propagation and Estimation]

Thank you for this suggestion. As in prior DICE methods, our duality results characterize the population problem, and we agree that the paper should make explicit where the neural implementation departs from it and how the resulting errors propagate. We have revised Appendix E accordingly.

1. Finite-data evidence already in the paper. Not all of our results are population-level. Proposition 2 (Proposition 4 in Appendix C.1) is a finite-data statement: the reward of the rarely observed action is estimated from a single noisy sample, and the gap arises from this sampling noise. A joint coefficient weak enough to move visitation toward the poorly covered state also makes the policy trust the noisy estimate, whereas separate coefficients avoid this trade-off. The Fourrooms study likewise uses 1,000 independently sampled finite buffers with estimated transitions, and the advantage persists when the transition estimator is changed (949 vs. 469 and 898 vs. 427 successes; Table 13). All benchmark results are reported with standard errors over eight seeds (five for LfO).

2. Where approximation enters (Appendix E). The algorithm never represents the occupancy explicitly. It minimizes the dual over (V,λ)(V, \lambda), which is convex in these functions, and the divergences enter only through conjugates f∗f^* evaluated as sample averages, so no divergence is estimated directly. Three approximations remain. (i) The action expectation inside the outer conjugate, which is specific to the decoupled dual: a single-sample plug-in would be biased by Jensen's inequality, so we estimate it with an auxiliary regression (Eq. 123) and use the chain-rule loss (Eq. 125), whose gradient equals that of the training objective under exact conditional estimates. (ii) Transition expectations, which are exact in our deterministic benchmarks and can be handled by a state–action auxiliary estimator otherwise. (iii) The initial-state term, approximated by a buffer average (Eqs. 119–120). Ent-SUNDER additionally uses a fixed histogram estimate of u/q^Bu/\hat q_B (Eqs. 128–129), and LfO-SUNDER (KL) a pretrained transition discriminator (Eqs. 144–145).

3. Error propagation for (i). Let e^d^\hat e_{\hat d} be the learned auxiliary function and ed^e_{\hat d} the exact conditional mean. Both Soft-chi conjugates in Table 14 have 1-Lipschitz derivatives, so ∣w^d(s)−wd(s)∣≤∣e^d^(s)−ed^(s)∣/β|\hat w_d(s) - w_d(s)| \le |\hat e_{\hat d}(s) - e_{\hat d}(s)|/\beta, and for deterministic dynamics the bias of the dual gradient satisfies

\big\\|\nabla L^{V,\lambda}_{\mathrm{Opt}} - \nabla J_{\mathrm{Opt}}\big\\| \\;\le\\; \frac{1}{\beta}\\,\big\\|\hat e_{\hat d} - e_{\hat d}\big\\|_{L^2(\hat q_B)}\\,\Big(\mathbb{E}_B\big\\|\nabla\lambda(s) + w_\pi(s,a)\\,\nabla e_\pi(s,a)\big\\|^2\Big)^{1/2}.

The gradient bias is therefore linear in the regression error of the auxiliary estimator and vanishes with it. Policy extraction uses only wπw_\pi, which is computed directly from (V,λ)(V, \lambda) and satisfies ∣w^π−wπ∣≤∣e^π−eπ∣/α|\hat w_\pi - w_\pi| \le |\hat e_\pi - e_\pi|/\alpha. We have added this analysis at the end of Appendix E.1.

4. Scope. A finite-sample guarantee under function approximation, such as a suboptimality bound under realizability and concentrability assumptions, is beyond the scope of this work, and we state this alongside the new analysis.

[W3. Theoretical assumptions and empirical estimation]

Thank you for raising this. The conditions fall into groups that differ in what can be checked in practice, so we address them separately.

1. Conditions on the generator hold by construction. Properness, closedness, and convexity are properties of the chosen function ff, not quantities estimated from data, so sampling cannot violate them. We use KL and Soft-chi (Table 14). Soft-chi is specified through its conjugate, which is finite, convex, and continuously differentiable; in primal form, f(u)=ulog⁡u−u+1f(u) = u \log u - u + 1 for 0≤u<10 \le u < 1 and f(u)=c(u−1)2f(u) = c (u-1)^2 for u≥1u \ge 1, with c=1c = 1 or c=1/2c = 1/2 depending on the instantiation. Hidden convexity also needs less than these three properties: Proposition 3 uses only convexity of ff and joint concavity of GG. Properness and closedness are used only in the conjugate-based steps, namely a well-defined conjugate and the recovery of density ratios from it.

2. Hidden convexity does not depend on how the reference distributions are obtained. Proposition 3 rests on two facts. First, for convex ff the policy regularizer is the perspective of ff in occupancy coordinates and is jointly convex in (ρ,d^)(\rho, \hat d) (Appendix B.1). Second, every GG in Table 1 is a sum of negative ff-divergences of affine functions of (d^,d′)(\hat d, d'), possibly plus linear terms, and is therefore jointly concave. Neither fact uses anything about the reference distributions beyond nonnegativity. Replacing a reference qq by an empirical distribution q^\hat q therefore gives the same convex program defined on q^\hat q. For example,

D^f(p∣q^)=∑xq^(x)f(p(x)q^(x))\hat D_f(p \\| \hat q) = \sum_x \hat q(x) f\left(\frac{p(x)}{\hat q(x)}\right)

remains convex in pp. The requirement p≪q^p \ll \hat q is not an additional condition to verify. KL and Soft-chi grow superlinearly, so the divergence is infinite whenever pp places mass outside the support of q^\hat q, and the recovered distributions are reweightings wq^w \hat q of the data with w≥0w \ge 0 (Eq. 18).

Three instantiations carry an extra, data-dependent assumption, which we state where it is used. Ent-SUNDER assumes that the data cover the uniform reference region (Appendix A.3). SAFE-SUNDER assumes that the offline state distribution contains the imperfect-data distribution as an ω\omega-fraction, so that the preferred-state reference is nonnegative (Appendix A.4). The mixed LfO reduction assumes the compatibility condition in Eq. 136.

3. The regularity conditions for strong duality concern the problem, not the samples. Weak duality holds without any assumption. For a zero duality gap and dual attainment, Appendix B.1 adds Slater-type conditions to the convexity established in Proposition 3: a finite primal optimum, a strictly positive feasible occupancy ρ∘(s,a)>0\rho^\circ(s,a) > 0, (d^∘,d′∘)∈ri⁡(dom⁡(−G))(\hat d^\circ, d'^\circ) \in \operatorname{ri}(\operatorname{dom}(-G)), positive reference-policy support πref(a∣s)>0\pi_{\mathrm{ref}}(a \mid s) > 0, and a finite local objective on the feasible support. These are conditions on the MDP and the reference distributions, and they are sufficient rather than necessary. They mirror the assumptions in prior DICE work: OptiDICE (Lee et al., 2021) assumes a strictly convex, continuously differentiable ff and dD>0d^D > 0, and its Supplementary Lemma 6 obtains a strictly feasible point from the assumption that all states are reachable. We agree that such conditions cannot be verified from finite data in continuous domains; this limitation is shared with prior DICE methods.

4. What the theory does not cover. The results above characterize the population (or tabular) problem. The practical algorithm adds function approximation and auxiliary estimators for the nested expectations. Appendix E states where the implemented gradient is exact only with exact conditional estimates, where we rely on deterministic dynamics, and where the initial-state term is approximated. This is the same kind of gap as in OptiDICE, whose sample-based objective is exact for deterministic MDPs and an upper bound otherwise (its Corollary 3). We do not claim finite-sample guarantees. As empirical evidence that the formulation behaves as intended when every quantity is estimated from data, the tabular Fourrooms study uses 1,000 independently sampled buffers with transitions estimated from each buffer, and the benefit of separate regularization persists under two transition estimators (949 vs. 469 and 898 vs. 427 successes out of 1,000; Table 13).

[Q1. Real-world experiments]

We have not run real-world experiments. This paper is a methodological contribution to the DICE family, and its evaluation follows the simulation-based protocol of prior DICE methods (Lee et al., 2021; Kim et al., 2022a; Lee et al., 2025).

There is also a reason specific to our claim. Testing whether state and policy regularization act independently requires knowing the quality of the dataset's states and actions, as well as the optimal visitation. Our controlled settings provide this ground truth: the state–action mismatch mazes (Section 5.1) and the truncated-coverage Fourrooms study (Section 3.1) are controlled versions of the partial or truncated data collection that motivates the paper (Ubellacker et al., 2024; Spencer et al., 2020).

Among the standard benchmarks, the Adroit human datasets consist of human demonstrations and are the closest proxy we have to real data collection. There, LfO-SUNDER attains the highest mean on two of the three tasks (Table 10), and Opt-SUNDER's largest margin over the baselines is on pen-human (75.03 vs. at most 53.86; Table 8). We do not regard this as a substitute for real-robot validation.

Reviewer_F1e2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that DICE methods regularize the joint state-action occupancy with a single coefficient, which ties together how far the learned policy may drift from the data in state visitation and in action selection. The authors propose SUNDER, which writes the occupancy as a state distribution times a policy and regularizes each with its own coefficient. They show the resulting problem is still convex (via a reparameterization), so the usual DICE dual derivation goes through, and instantiate it for reward maximization, state-entropy maximization and learning from observations, recovering several existing DICE methods as the case of equal coefficients. Experiments include toy mazes where state and action quality are controlled separately, Minari/D4RL benchmarks, entropy maximization on mazes and DMC, and LfO on MuJoCo and Adroit.

Critical strengths
  • The paper identifies a realistic failure mode of joint occupancy regularization: offline data where state coverage and action quality are mismatched (good actions at suboptimal states, or good states with suboptimal actions). It argues convincingly that a single joint coefficient forces an unnecessary trade-off in such cases.
  • Extending DICE to allow separate state and policy regularization strengths is a natural solution that retains the tractable dual form of DICE methods.
  • While it can be presented more clearly, the maze setting with independently controlled state and action quality (Sec. 4.1) seems to isolate the intended effect well.
  • The idea is instantiated beyond reward maximization, to state-entropy maximization and learning from observations, suggesting it is not tied to a single objective.
  • As far as I checked, the derivations in the paper appear correct.
Critical weaknesses
  • To the best of my understanding, Proposition 1 and the resulting dual derivations are standard perspective-function and Fenchel arguments, with limited technical novelty.

  • Theorems 1 and 2 show that different (α, β) yield different optima than a single joint coefficient, which is close to tautological. They do not show that the decoupled optimum is better for some real settings.

  • Experiments (main weakness). The experimental methodology is the paper's main weakness. The setup is underspecified (no pseudo-code, no description of which networks are trained, when, or with which f-divergence); tuning is asymmetric (OptiDICE sweeps 5 values of α with a single seed, while Opt-SUNDER is tuned per task on a 2D grid, and Fourrooms picks the best coefficients per instance, which favors a 2D search by construction); the α = β sanity check fails (with KL, α = β should recover OptiDICE, yet Table 2a differs at α = β = 0.1: 53.9 vs. 50.7 and 42.4 vs. 45.6), suggesting implementation confounds; much of the Minari gain comes from the flow extractor, not decoupling (OptiDICE goes from 89.4 to 113.3 with flow, vs. 113.5 for Opt-SUNDER on MuJoCo); Ent-SUNDER outperforming SEMDICE is close to built in, since it drops the low-entropy state regularizer; and the tuning procedure and dataset construction for the non-DICE baselines are unclear. I note that addressing these issues would essentially require redoing the empirical evaluation, which I doubt is feasible within the rebuttal period.

Theory: presentation (minor)

  • G is introduced loosely, and it is never stated plainly that G absorbs the state-distribution regularization. That absorption is the crux of the "decoupling" and should be explicit.
  • Table 1 should list the final objective actually optimized for each instance. Currently this is recoverable only from Appendix A.
  • Section 3.6 is largely standard (weighted BC/CFM). Only the ratio-recovery equation is specific to SUNDER.
  • The Fourrooms motivation gets its own subsection, while the more relevant argument for decoupling (Appendix C) is relegated to the appendix. Consider swapping them.

Overall presentation

  • There is no related-work section. In particular, the paper does not discuss prior work distinguishing state-distribution regularization and action/policy regularization in RL.
  • There is no algorithm box, and implementation details are missing.
  • There are signs of rushed preparation: a "Table X" placeholder (App. D.1), and an unclear Figure 2.
Questions for authors
  1. When is decoupling beneficial in practice? Outside the hand-crafted maze settings, it is unclear whether the motivating mismatch (good actions at poorly covered states, or vice versa) occurs in standard datasets. Can the authors suggest a way to detect it from the offline data, i.e., predict when separate regularization will help? Relatedly, what is SUNDER's overhead relative to OptiDICE (additional networks, wall-clock time), and when is the extra tuning coefficient worth it?

  2. Coverage-based perspective. The paper could benefit from a result in the language of coverage (concentrability) coefficients, which are standard in offline RL analyses. The coefficient d^{π*}(s,a)/q_B(s,a) factorizes exactly into a state term d̂^{π*}(s)/q̂_B(s) and an action term π*(a|s)/π_B(a|s), mirroring the two shifts SUNDER controls separately. Can the authors show that SUNDER improves over joint regularization when one of these terms is small?

5 comments

[W1. Technical novelty.]

We agree that the proof tools are standard. In occupancy coordinates ρ = d̂π, the policy regularizer is a perspective function and the dual follows from Fenchel conjugacy, as Appendix B states. (Proposition 1 of the submission is Proposition 3 in the revision.) We do not claim a new convex-analysis technique. The contribution is what this observation makes possible, which to our knowledge is new for DICE:

  1. A decoupled dual. Existing DICE methods regularize the joint occupancy with one f-divergence, so a single coefficient governs both state and policy shift. In SUNDER, eliminating π produces a per-state term e_d̂(s) = λ(s) + α E_{π_ref}[f*(e_π/α)], which then enters the conjugate of the state objective (Eq. 6). This nested dual, with a multiplier for policy normalization and separate policy and state density ratios, does not arise under joint regularization.
  2. One derivation for many objectives. G can be any jointly concave function of (d̂, d′). The same steps give decoupled counterparts of six existing DICE methods and a new transition-level LfO formulation (Table 1).
  3. A trainable estimator. The nested dual places an expectation over actions inside an outer nonlinearity, which cannot be estimated from one sampled action per state. Appendix E gives the auxiliary regression and chain-rule gradient that handle this.

[W2. Why decoupling matters, and when.]

We agree with the main point. The non-recoverability results (Theorems 1 and 2 of the submission; Proposition 1 and Theorem 1 in the revision) show only that the second coefficient is not redundant: a decoupled optimum generally cannot be reproduced by re-tuning the joint coefficient. They do not show that it is better. The revision adds this.

Theory. Proposition 2 (Section 3.1; full statement and proof as Proposition 4, Appendix C.1) constructs MDPs and datasets with good action coverage (C_a* → 1) and poor state coverage (C_s* → ∞). Separate regularization has suboptimality at most (2+√e)/8 · (C_a* − 1), which vanishes. Joint regularization keeps an expected gap of 1/32, even when an oracle chooses the coefficient after seeing the data to maximize true return. Resisting a noisy reward estimate requires strong policy regularization, and a joint coefficient then also ties visitation to poorly covered states.

Experiments. The same holds empirically:

  • Fourrooms, with one coefficient setting shared across 1,000 random buffers: within 1% of optimal on 949 buffers, versus 469 for joint KL (Figure 1).
  • Controlled state–action mismatch (Table 2): 71.67 versus at most 54.32 for OptiDICE over the swept joint coefficients, and 53.66 versus at most 45.59 in the reverse setting.
  • Standard benchmarks under an equal search budget (Table 7): 114.68 versus 101.08 on AntMaze and 46.42 versus 39.85 on Adroit, with MuJoCo tied.
  • State-entropy maximization (Tables 4a and 9): Ent-SUNDER exceeds SEMDICE on all seven mazes, with average entropy 1.460 versus 0.837.

When it matters. In reward maximization, decoupling helps when state and action quality differ. It is more central to SEM and LfO, where changing state visitation is itself the objective: regularizing visitation toward the dataset then works against the objective, while policy regularization remains useful.

[W3 and W9. Experiments, algorithm, and implementation details]

We thank the reviewer for the detailed assessment. We were able to address the main experimental concerns within the rebuttal period. The revision adds an equal-budget comparison on all 27 Minari tasks with eight seeds (Table 7), a Fourrooms study rerun with one coefficient setting shared across all 1,000 buffers (Figure 1, Table 13), an extractor-controlled comparison (Tables 3 and 6), and an algorithm box with architectures and divergence settings (Appendix E). We answer the six points of W3 in order.

(a) "The setup is underspecified" (with W9)

Appendix E now contains the training procedure (Algorithm 1), the divergence choices (Table 14), the network architectures (Table 15), and the losses and update order of each method (Appendices E.1–E.4). In short, Opt-SUNDER and Ent-SUNDER train VV, λ\lambda, and an auxiliary state function for the inner action expectation, all updated at every iteration. These functions are then frozen, and the policy is trained by ratio-weighted flow matching. Opt-SUNDER uses Soft-chi for both the state and the policy regularizer, and OptiDICE uses the same Soft-chi generator.

(b) "Tuning is asymmetric"

Equal budget. Both methods now select one coefficient per task from the same three candidates, 0.25,0.5,1.0\\{0.25, 0.5, 1.0\\}. OptiDICE tunes its joint coefficient. Opt-SUNDER tunes only the state coefficient β\beta, with the policy coefficient fixed at α=1\alpha = 1 for all tasks, so both searches are one-dimensional. We fix α\alpha rather than β\beta because out-of-distribution actions are the more critical failure mode in offline RL. The selection procedure is the same for both methods: for each task, the three candidates are screened on one or two seeds (on some tasks, on all eight), and the best candidate is then run with eight seeds, which include the screening seeds.

DomainTasksOpt-SUNDEROptiDICE
MuJoCo9113.53 ± 0.39113.31 ± 0.42
AntMaze6114.68 ± 1.84101.08 ± 1.72
Adroit1246.42 ± 0.1639.85 ± 1.24
Overall2783.96 ± 0.4977.94 ± 0.73

Mean ± SE over eight seeds (Table 7, Appendix D.4). The SE is computed across seeds of the domain average.

The two methods are tied on MuJoCo, and the gain remains on AntMaze and Adroit. Two further results indicate that it does not come from tuning capacity. Opt-SUNDER with three candidates is above OptiDICE with the five-value sweep of the main results (83.96 vs. 80.44 on the 27-task average). And when α\alpha and β\beta are set from dataset statistics by a rule that uses no per-task evaluation feedback, Opt-SUNDER reaches 84.01 (Table 12, Appendix D.7).

Fourrooms. We replaced per-instance tuning with one coefficient setting per method, shared across all 1,000 buffers. A shared setting cannot adapt to individual buffers, so the comparison no longer rewards a larger per-instance search. Separate regularization is within 1% of the optimal return on 949 buffers, compared with 469 for joint regularization (Figure 1, Appendix D.8). With sample-based transition estimates instead of an estimated transition matrix, the counts are 898 and 427 (Table 13).

(c) "The α = β sanity check fails"

The sanity check holds for KL. With α=β=η\alpha = \beta = \eta, substituting λ⋆(s)=ηlog⁡E[exp⁡(A(s,a)/η−1)]\lambda^{\star}(s) = \eta \log \mathbb{E}[\exp(A(s,a)/\eta - 1)] into the Opt-SUNDER dual gives exactly the OptiDICE dual (Eq. 4). Here the expectation is over dataset actions at ss, and A(s,a)=r(s,a)+γ,E[V(s′)]−V(s)A(s,a) = r(s,a) + \gamma\\,\mathbb{E}[V(s')] - V(s). We have added this derivation to Appendix A.2.

Table 2 does not use KL. Both methods use the same Soft-chi generator, and the Soft-chi divergence of the joint occupancy does not split into a state term plus a policy term. Equal coefficients therefore give two different objectives, and the results need not match. The differences at α=β=0.1\alpha = \beta = 0.1 are also not systematic: Opt-SUNDER is higher by 3.2 in one setting (53.9 vs. 50.7) and lower by 3.2 in the other (42.4 vs. 45.6). An implementation advantage would appear with the same sign in both.

(d) "Much of the Minari gain comes from the flow extractor"

We agree. Most of the gain over Gaussian-policy OptiDICE comes from the flow extractor, which is why OptiDICE (Flow) is the controlled baseline. Appendix D.2 now states this explicitly.

DomainOptiDICE (Gaussian)OptiDICE (Flow)Opt-SUNDER (Flow)
MuJoCo89.43 ± 2.57113.31 ± 0.40113.84 ± 0.42
AntMaze18.76 ± 1.50103.04 ± 1.56117.26 ± 1.50
Adroit34.85 ± 0.5444.49 ± 0.8548.24 ± 0.25

Mean ± SE over eight seeds (Table 3). Domain SEs are propagated from task-level SEs (Eq. 118).

The effect of decoupling is the difference between the last two columns: none on MuJoCo (within one standard error of the difference), +14.2 on AntMaze, and +3.8 on Adroit. It persists under the equal budget in (b), at +13.6 and +6.6. We have revised the Introduction and Section 5.1 so that they no longer claim an improvement on MuJoCo.

(e) "Ent-SUNDER outperforming SEMDICE is close to built in"

We agree that the direction of the effect follows from the objective. The experiment tests whether it can be realized from offline data. SEMDICE has a single coefficient, so weakening its state regularization also weakens its policy regularization. Ent-SUNDER removes only the former. Ent-SUNDER has the higher mean entropy on all 11 tasks. The margin is clear on the seven mazes (average 1.460 vs. 0.837) and on Acrobot, and within two standard errors on Reacher, Point mass, and Pendulum (Tables 4a and 9).

(f) "Tuning procedure and dataset construction for the non-DICE baselines are unclear"

Baselines (Appendix D.1, p. 29, lines 1527–1531):

  • Reward maximization: we run the baselines with the official GFP implementation (https://github.com/Simple-Robotics/guided-flow-policy) on seeds 1–8, with the hyperparameters released with the code. When GFP's settings differ from a baseline's official settings, we use the official settings or evaluate both and report the better result. For example, ReBRAC (https://github.com/tinkoff-ai/ReBRAC) improves from 34.96 to 71.57 on AntMaze with its official configuration, and we report 71.57. We did no further tuning.
  • SEM: the official APT and PROTO agents from URLB (https://github.com/rll-research/url_benchmark).
  • LfO: official implementations and released configurations, except LobsDICE, which we implemented following its paper.

Datasets (Appendix D.1, p. 29, lines 1520–1526):

  • Reward maximization: the official Minari datasets (https://minari.farama.org/) for all 27 tasks.
  • Controlled mazes: six of the seven ExDM mazes (https://github.com/yingchengyang/ExDM), namely square-a, square-b, square-c, square-d, tree, and large; only bottleneck is excluded. The optimal policy is computed analytically. Poor-state, good-action data come from epsilon-greedy rollouts of the optimal policy started from poor states. Good-state, poor-action data come from good states, using an epsilon-greedy policy directed opposite to the optimal policy.
  • SEM: we collected DMC data and removed part of it to obtain datasets with low state entropy.
  • LfO: the datasets and data configurations released with IOSTOM (https://github.com/quanganh1999/IOSTOM).

[W4–W8, W10, and W11. Presentation and organization]

Thank you for these suggestions. We have revised the paper as follows.

CommentRevisionLocation
W4. Role of GGStated explicitly that GG contains the state-distribution regularization, with reward maximization as the example.Section 3.2
W5. Final objectivesAdded the final dual objective of each instantiation. The sample-based losses are in Appendix E.Table 1(b), p. 7
W6. Policy extractionShortened the standard BC/CFM discussion and kept the ratio-recovery equation.Section 3.6
W7. Theory before FourroomsMoved the theoretical motivation (Propositions 1 and 2) ahead of the Fourrooms illustration.Section 3.1
W8. Related workAdded a Related Work section. It covers Neu et al. (2017), who distinguish joint state–action regularization from conditional policy regularization, and Geist et al. (2019) on policy regularization.Section 4
W10. "Table X"Corrected the table reference.Appendix D
W11. Figure 2Rewrote the caption.Figure 2
Reviewer_gqCsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

DICE methods optimize state-action occupancies but couple state visitation and action selection through one regularizer. SUNDER separates them: the state marginal d^\hat d is regularized toward a reference state distribution while the policy π\pi is regularized toward πref\pi_{\text{ref}}. The resulting formulation is bilinear, but the paper proves a hidden convex structure (Proposition 1) that restores strong duality and yields a general dual in which π\pi and the occupancy variables are eliminated analytically. Six existing DICE variants are recovered as special cases, and new instantiations are proposed for reward maximization, state-entropy maximization and learning from observations, together with a flow-matching policy extractor that uses the density ratio as a sample weight. Experiments cover controlled mazes, 27 Minari tasks, state-entropy domains and LfO on MuJoCo/Adroit.

Critical strengths
  1. The separation is not cosmetic: the paper identifies why naive decoupling breaks the standard affine argument, then proves convexity of the value set under a specific occupancy-weighted mixture and derives a general dual. Recovering OptiDICE, SEMDICE, SMODICE, DemoDICE, SafeDICE and LobsDICE as special cases is strong evidence that the framework is the right generalization.

  2. The controlled experiments target the actual claim. In Fourrooms, separate regularization attains a 1% optimality gap in 958/1000 buffers versus 526 for joint regularization, and the two constructed maze settings (good states/bad actions and vice versa) show the intended asymmetric response. The flow-matching extractor also removes the guidance-scale hyperparameter of Diffusion-DICE.

Critical weaknesses
  1. The convexity result is elegant but its practical reach is unclear. It requires GG jointly concave and establishes duality for the population problem; the algorithm still solves a nonconvex saddle point with extra dual variables λ(s)\lambda(s) and Q(s,s′)Q(s,s'), and Q(s,s′)Q(s,s') scales with ∣S∣2|S|^2 for transition-level objectives. The paper does not say how these are parameterized, whether the dual is attained, or how the construction behaves when d^θ(s)=0\hat d_\theta(s)=0.

  2. The empirical case for decoupling is strongest exactly where it is easiest. The maze studies use an estimated transition matrix and small state spaces, while on standard Minari/MuJoCo benchmarks the gains may fall within seed variance. Decoupling also introduces two regularization strengths instead of one, yet no matched tuning budget is reported, so "best average" may partly reflect extra tuning capacity.

Questions for authors
  1. How are λ(s)\lambda(s) and Q(s,s′)Q(s,s') parameterized and optimized, and what is the cost for transition-level objectives?
  2. Does strong duality survive the sample-based estimator, and what happens on states with d^(s)=0\hat d(s)=0?
  3. With an equal tuning budget, does Opt-SUNDER still beat OptiDICE on the 27 Minari tasks? Please report seed variance and significance.
  4. How much of the maze gain depends on having an estimated transition matrix?
  5. How does LfO-SUNDER compare with state-matching baselines such as SMODICE under matched tuning?
6 comments

[W1&Q1-2,4 Practical reach of the convexity result]

Hidden convexity is what makes the decoupled problem usable in practice: it replaces a primal that requires the transition dynamics by a single minimization over dual functions estimated from offline samples, as in standard DICE.

(a) "requires jointly concave GG ... duality for the population problem"

Joint concavity is not an assumption added by decoupling: standard DICE relies on the same concavity of its occupancy objective. It holds for every instantiation in Table 1, where each GG is a sum of negative ff-divergences of affine functions of (d^,d′)(\hat d, d').

Does duality survive sampling? Strong duality is a property of the optimization problem. Proposition 3 holds for any finite MDP, so it applies unchanged to the plug-in problem built from an estimated transition matrix. For the neural implementation, what is estimated from samples is the dual objective; we do not claim finite-sample guarantees, as in DualDICE, OptiDICE, and SMODICE. Appendix E states where approximations enter.

Empirically, over 1,000 finite tabular buffers, separate regularization is within 1% of optimal on 949 versus 469 for joint KL with an estimated transition matrix, and on 898 versus 427 with sample-based transition expectations (Table 13).

(b) "nonconvex saddle point ... how these are parameterized"

The final problem is not a saddle point. After the closed-form elimination of (d^,d′,π)(\hat d, d', \pi) in Section 3.4, the dual in Eq. (6) is a single minimization over (V,λ,Q)(V, \lambda, Q), with no inner maximization or adversarial training. In function space it is convex, as a pointwise supremum of a Lagrangian that is affine in the dual variables, with zero duality gap under the conditions of Appendix B.1 (Proposition 3). The Opt-, Ent- and KL-LfO objectives we implement inherit this convexity; the mixed LfO objective (Eq. 97) coincides with the dual only under the compatibility condition of Eq. 138, as noted in Table 1, and need not be convex otherwise. The auxiliary networks in Appendix E are regression estimators of conditional expectations, not adversaries, and the KL-LfO transition discriminator is pretrained and then frozen. Beyond this, nonconvexity arises only from the neural parameterization, as in OptiDICE.

Parameterization and optimization (Table 15, Algorithm 1). λ(s)\lambda(s) is a state network with the same architecture as VV, and Q(s,s′)Q(s,s') is a network on the concatenated pair. Each iteration takes a gradient step on the dual networks and a regression step on the auxiliary estimators; optimizer settings are listed with Table 15. The two multipliers never appear together in the evaluated methods: Opt-/Ent-SUNDER use (V,λ)(V, \lambda), and LfO-SUNDER uses (V,Q)(V, Q) (Appendix A.8).

Cost for transition-level objectives. QQ is evaluated only on sampled transitions, so computation scales with the minibatch size and does not require enumerating ∣S∣2|\mathcal S|^2 state pairs. On HalfCheetah-medium-v2 with a single RTX PRO 6000 Blackwell GPU, 500k DICE updates of LfO-SUNDER (KL variant) take 369 s, excluding discriminator pretraining and policy extraction. For reference, the state-level Opt-SUNDER takes 437.8 s, 45.6% more time and 17.2% more peak GPU memory than OptiDICE with identical network sizes (Appendix D.5).

(c) "whether the dual is attained ... when d(s)=0d(s)=0"

Dual attainment. For finite state and action spaces, Appendix B.1 gives standard Slater-type conditions: a strictly feasible occupancy (e.g., when all states are reachable) and sufficient support of the reference distributions. Under these conditions strong duality holds, as in the Slater-based argument of OptiDICE (Lee et al., 2021, Supplementary Lemma 6), and the dual optimum is attained whenever the primal value is finite.

Zero-occupancy states. The construction remains well defined when d^(s)=0\hat d(s)=0. In Eqs. (5a)–(5d), every term containing π(⋅∣s)\pi(\cdot\mid s) is multiplied by d^(s)\hat d(s), so at such a state π\pi affects neither feasibility nor the objective. The proof of Proposition 3 covers this case: if d^θ(s)=0\hat d_\theta(s)=0 for θ∈(0,1)\theta\in(0,1), then d^1(s)=d^2(s)=0\hat d_1(s)=\hat d_2(s)=0 and πθ(⋅∣s)\pi_\theta(\cdot\mid s) can be chosen arbitrarily. For policy extraction, π∗\pi^{\ast} is identified only at states with positive target occupancy (Section 3.6). With the KL and Soft-chi generators we use, (f∗)′>0(f^{\ast})' > 0, so the recovered state ratio wd∗w_d^{\ast} is strictly positive on the offline state support; zero-occupancy states can arise only outside the data, where the extraction loss has no samples.

Revised (in blue). Table 15 and Algorithm 1 (parameterization, optimizer); Appendix A (concavity of GG); A.8 (mixed-LfO scope); B.1 and A.2 (scope of the dual, d^(s)=0\hat d(s)=0); D.5 (LfO timing); Section 3.6 (π∗\pi^{\ast}).

(c) "whether the dual is attained, or how the construction behaves when d(s)=0d(s)=0"

Dual attainment. For finite state and action spaces, Appendix B.1 gives standard Slater-type conditions: a strictly feasible occupancy (e.g., when all states are reachable) and sufficient support of the reference distributions. Under these conditions, strong duality holds and the dual optimum is attained whenever the primal value is finite. A similar Slater-based argument is used in OptiDICE (Lee et al., 2021, Supplementary Lemma 6).

Zero-occupancy states. The construction remains well defined when d^(s)=0\hat d(s)=0. In Eqs. (5a)–(5d), π(⋅∣s)\pi(\cdot\mid s) enters the objective and every constraint only through the product d^(s)π(a∣s)\hat d(s)\pi(a\mid s), so at such a state it affects neither feasibility nor the objective. The proof of Proposition 3 covers this case: if d^θ(s)=0\hat d_\theta(s)=0 for θ∈(0,1)\theta\in(0,1), then d^1(s)=d^2(s)=0\hat d_1(s)=\hat d_2(s)=0, every π\pi-dependent term at ss vanishes on both sides, and πθ(⋅∣s)\pi_\theta(\cdot\mid s) can be chosen arbitrarily. In the occupancy coordinates of Appendix B.1, this is the standard closure of the perspective function, 0⋅f(0/0)=00\cdot f(0/0)=0. The dual derivation is unaffected, since the maximization over π(⋅∣s)\pi(\cdot\mid s) is multiplied by d^(s)≥0\hat d(s)\ge 0. For policy extraction, π∗\pi^* is identified only at states with positive target occupancy, as now stated in Section 3.6. With the KL and Soft-chi generators we use, (f∗)′>0(f^*)' > 0, so in Opt-SUNDER the recovered state ratio wd∗w_d^* is strictly positive on the offline state support; zero-occupancy states can arise only outside the data, where the extraction loss has no samples.

[W2 and Q3. Empirical evidence, tuning budget, and significance]

We thank the reviewer for these comments and answer W2 and Q3 together, since they concern the same evidence. We agree with part of the assessment: on MuJoCo the gain is within seed variance, and we have revised the paper so that it no longer claims an improvement there. On AntMaze and Adroit, however, the gains are well outside seed variance, hold under an equal search budget, and hold without any per-task tuning. Our claim is not that decoupling improves performance universally, but that it helps when state and policy call for different regularization strengths.

(a) W2: "The maze studies use an estimated transition matrix and small state spaces"

The paper contains two different maze studies. Fourrooms (Figure 1, Appendix D.8) is a tabular illustration and does use a transition matrix estimated from the buffer. Its role is to isolate separate versus joint regularization over 1,000 independently generated buffers, and the advantage persists when the matrix is replaced by sample-based transition estimates (898 vs. 427 successes; Table 13). The controlled mazes of Section 5.1 are different: they use the ExDM maze environments and are trained with the neural-network implementation of Opt-SUNDER, with no transition model, as now detailed in Appendices D.1 and E.1. These mazes are small by design, since this is what allows state quality and action quality to be set independently. They test the mechanism; scalability is tested on the Minari benchmarks.

(b) W2: "no matched tuning budget is reported"; Q3: "With an equal tuning budget, does Opt-SUNDER still beat OptiDICE?"

Yes on the 27-task average (Table R1). Both methods select one coefficient per task from the same three candidates, 0.25,0.5,1.0\\{0.25, 0.5, 1.0\\}. OptiDICE tunes its single joint coefficient. Opt-SUNDER tunes only the state coefficient β\beta, with the policy coefficient fixed at α=1\alpha = 1 for all tasks; we keep policy regularization strong because out-of-distribution actions are the more critical failure mode in offline RL. For each task, the three candidates are first screened on one or two seeds (on some tasks, on all eight), and the best candidate is then run with eight seeds, which include the screening seeds. Both methods follow this procedure.

Table R1. Normalized scores (mean ± SE over eight seeds). "Candidates" is the number of configurations searched per task. The last two rows are reference results from the paper.

MethodCandidatesMuJoCo (9)AntMaze (6)Adroit (12)Overall (27)
OptiDICE3113.31 ± 0.42101.08 ± 1.7239.85 ± 1.2477.94 ± 0.73
Opt-SUNDER3113.53 ± 0.39114.68 ± 1.8446.42 ± 0.1683.96 ± 0.49
OptiDICE (main results, Tables 3 and 12)5113.31 ± 0.40103.04 ± 1.5644.49 ± 0.8580.44
Opt-SUNDER (dataset-statistics rule, Table 12)none113.14 ± 0.47115.49 ± 1.6446.42 ± 0.2284.01

The gain also does not depend on per-task search. With α\alpha and β\beta set from dataset statistics by a rule fixed before evaluation (Appendix D.7), Opt-SUNDER reaches 84.01 on the 27-task average, above OptiDICE with a five-value per-task sweep (80.44; last two rows). The best average is therefore not explained by additional tuning capacity.

(c) W2: "the gains may fall within seed variance"; Q3: "Please report seed variance and significance"

Table R2 reports two tests for the equal-budget comparison (first two rows of Table R1). The seed-level test asks whether the difference exceeds seed variance (Welch's t-test on the domain means, using the seed-based standard errors of Table R1). The across-task test asks whether the difference is consistent across tasks (paired t-test on per-task means).

Table R2. Opt-SUNDER minus OptiDICE under the equal budget.

DomainTasksDifference ± SESeed-level pAcross-task 95% CIAcross-task p
MuJoCo9+0.22 ± 0.570.71[−0.48, 0.92]0.49
AntMaze6+13.60 ± 2.52< 0.01[4.79, 22.41]0.011
Adroit12+6.57 ± 1.25< 0.01[−1.51, 14.65]0.10
Overall27+6.02 ± 0.88< 0.001[1.92, 10.11]0.006

On MuJoCo the reviewer is correct: the two methods are tied, both here and in the main results (113.84 ± 0.42 vs. 113.31 ± 0.40; Table 3), and we have revised the Introduction and Section 5.1 accordingly. On AntMaze, Adroit, and the 27-task average, the gains are significant relative to seed variance, as they are in the main results (+14.2 and +3.8 on AntMaze and Adroit, about 6.6 and 4.2 standard errors of the difference). Across tasks, the gain is significant on AntMaze and overall but not on Adroit (p = 0.10): there it is concentrated in a subset of tasks, while on several tasks both methods are either near zero or near the maximum score (Table 8). The main results show the same pattern: Opt-SUNDER is better than OptiDICE by more than two standard errors and at least one point on 8 of the 27 tasks, worse on 2, and tied on 17, with the wins concentrated on AntMaze (4 of 6 tasks) and on pen-expert, pen-human, and door-human. The gains therefore appear on sparse-reward navigation and dexterous manipulation rather than on the easiest tasks, consistent with our claim that decoupling helps where joint regularization is restrictive and is neutral where it already suffices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.