- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Classical DICE algorithms apply joint state visitation and action selction for constructing occupancy regularization. In this mannuscript, the authors propose SUNDER (State policy UNcoupling for Distribution Estimation via Ratio factorization) to seperately contril state visitation and action selection. The authors shows the appealing properties of SUNDER such as hidden convexity and strong duality preserving. Evaluated on Maze, Minari / D4RL benchmarks, SUNDER methods (Opt‑SUNDER, Ent‑SUNDER, LfO‑SUNDER) fully demonstrate the superiority.
Critical strengths
-
The studied problem is interesting and critical. State‑distribution shift and policy action deviation are two distinct dimensions, and the authors point out an important issue to study.
-
The theoretical contribution of the proposed method is appealing. The paper reveals hidden convexity via perspective‑convexity, preserving strong duality.
-
The manuscript is well-organized, and the proposed method is one unified framework for implementing various tasks: reward maximization, state‑entropy maximization, learning‑from‑observations.
Critical weaknesses
-
Compared to vanilla DICE, SUNDER optimizes dual objective for both state and policy regularizers, which leads to extra complexity. The paper does not report runtime or GPU memory usage, lacking quantitative computational‑cost comparison against OptiDICE and other baselines.
-
All theoretical hidden‑convexity and non‑recoverability theorems are hold at population level, assuming exact occupancy representation. However, we use neural‑network for implementation, which suffers function‑approximation error, sampling noise and empirical f‑divergence estimation bias. It is suggested to provide further analysis on practical error propagation and estimation.
-
The theoretical prerequisites are somehow strong and hard to verify in practice. For instance, hidden convexity and strong duality require proper, closed, convex f‑divergence generator. However, empirical sample‑estimated f‑divergence can violate conditions.
Questions for authors
-
The experiment studies are performed on Minari / D4RL simulation; dataset. Is there any real-scenario experiment?
-
The performance of LFO-SUNDER is the worst among all competitors under MuJoCo tasks, any explanations on this issue?
-
It seems the performance of SUNDER depends on policy extractor capacity, while Flow‑Matching extractor is much better than guassian. Any ablation on how to select policy extractor, especially in cases where policy class lacks sufficient capacity to fit target density ratios.
5 comments
3. Error propagation for (i). Let be the learned auxiliary function and the exact conditional mean. Both Soft-chi conjugates in Table 14 have 1-Lipschitz derivatives, so , and for deterministic dynamics the bias of the dual gradient satisfies
\big\\|\nabla L^{V,\lambda}_{\mathrm{Opt}} - \nabla J_{\mathrm{Opt}}\big\\| \\;\le\\; \frac{1}{\beta}\\,\big\\|\hat e_{\hat d} - e_{\hat d}\big\\|_{L^2(\hat q_B)}\\,\Big(\mathbb{E}_B\big\\|\nabla\lambda(s) + w_\pi(s,a)\\,\nabla e_\pi(s,a)\big\\|^2\Big)^{1/2}.
The gradient bias is therefore linear in the regression error of the auxiliary estimator and vanishes with it. Policy extraction uses only , which is computed directly from and satisfies . We have added this analysis at the end of Appendix E.1.
4. Scope. A finite-sample guarantee under function approximation, such as a suboptimality bound under realizability and concentrability assumptions, is beyond the scope of this work, and we state this alongside the new analysis.
[W3. Theoretical assumptions and empirical estimation]
Thank you for raising this. The conditions fall into groups that differ in what can be checked in practice, so we address them separately.
1. Conditions on the generator hold by construction. Properness, closedness, and convexity are properties of the chosen function , not quantities estimated from data, so sampling cannot violate them. We use KL and Soft-chi (Table 14). Soft-chi is specified through its conjugate, which is finite, convex, and continuously differentiable; in primal form, for and for , with or depending on the instantiation. Hidden convexity also needs less than these three properties: Proposition 3 uses only convexity of and joint concavity of . Properness and closedness are used only in the conjugate-based steps, namely a well-defined conjugate and the recovery of density ratios from it.
2. Hidden convexity does not depend on how the reference distributions are obtained. Proposition 3 rests on two facts. First, for convex the policy regularizer is the perspective of in occupancy coordinates and is jointly convex in (Appendix B.1). Second, every in Table 1 is a sum of negative -divergences of affine functions of , possibly plus linear terms, and is therefore jointly concave. Neither fact uses anything about the reference distributions beyond nonnegativity. Replacing a reference by an empirical distribution therefore gives the same convex program defined on . For example,
remains convex in . The requirement is not an additional condition to verify. KL and Soft-chi grow superlinearly, so the divergence is infinite whenever places mass outside the support of , and the recovered distributions are reweightings of the data with (Eq. 18).
Three instantiations carry an extra, data-dependent assumption, which we state where it is used. Ent-SUNDER assumes that the data cover the uniform reference region (Appendix A.3). SAFE-SUNDER assumes that the offline state distribution contains the imperfect-data distribution as an -fraction, so that the preferred-state reference is nonnegative (Appendix A.4). The mixed LfO reduction assumes the compatibility condition in Eq. 136.
3. The regularity conditions for strong duality concern the problem, not the samples. Weak duality holds without any assumption. For a zero duality gap and dual attainment, Appendix B.1 adds Slater-type conditions to the convexity established in Proposition 3: a finite primal optimum, a strictly positive feasible occupancy , , positive reference-policy support , and a finite local objective on the feasible support. These are conditions on the MDP and the reference distributions, and they are sufficient rather than necessary. They mirror the assumptions in prior DICE work: OptiDICE (Lee et al., 2021) assumes a strictly convex, continuously differentiable and , and its Supplementary Lemma 6 obtains a strictly feasible point from the assumption that all states are reachable. We agree that such conditions cannot be verified from finite data in continuous domains; this limitation is shared with prior DICE methods.
4. What the theory does not cover. The results above characterize the population (or tabular) problem. The practical algorithm adds function approximation and auxiliary estimators for the nested expectations. Appendix E states where the implemented gradient is exact only with exact conditional estimates, where we rely on deterministic dynamics, and where the initial-state term is approximated. This is the same kind of gap as in OptiDICE, whose sample-based objective is exact for deterministic MDPs and an upper bound otherwise (its Corollary 3). We do not claim finite-sample guarantees. As empirical evidence that the formulation behaves as intended when every quantity is estimated from data, the tabular Fourrooms study uses 1,000 independently sampled buffers with transitions estimated from each buffer, and the benefit of separate regularization persists under two transition estimators (949 vs. 469 and 898 vs. 427 successes out of 1,000; Table 13).
[Q1. Real-world experiments]
We have not run real-world experiments. This paper is a methodological contribution to the DICE family, and its evaluation follows the simulation-based protocol of prior DICE methods (Lee et al., 2021; Kim et al., 2022a; Lee et al., 2025).
There is also a reason specific to our claim. Testing whether state and policy regularization act independently requires knowing the quality of the dataset's states and actions, as well as the optimal visitation. Our controlled settings provide this ground truth: the state–action mismatch mazes (Section 5.1) and the truncated-coverage Fourrooms study (Section 3.1) are controlled versions of the partial or truncated data collection that motivates the paper (Ubellacker et al., 2024; Spencer et al., 2020).
Among the standard benchmarks, the Adroit human datasets consist of human demonstrations and are the closest proxy we have to real data collection. There, LfO-SUNDER attains the highest mean on two of the three tasks (Table 10), and Opt-SUNDER's largest margin over the baselines is on pen-human (75.03 vs. at most 53.86; Table 8). We do not regard this as a substitute for real-robot validation.
We thank the reviewer for their insightful comments and have updated the manuscript accordingly.
[W1. Computational Cost of State–Policy Decoupling]
Thank you for pointing this out. We have added a computational-cost comparison to Appendix D.5 of the revised manuscript.
We compare against OptiDICE because it is the joint-regularization counterpart of Opt-SUNDER. The two methods use identical per-network sizes and the same policy-extraction procedure, so the comparison isolates the cost attributable to state–policy decoupling. The non-DICE baselines belong to a different algorithm family and are run from the official GFP implementation, so a wall-clock comparison with them would reflect those differences rather than the cost of decoupling. Both methods were trained on HalfCheetah on a single NVIDIA RTX PRO 6000 Blackwell GPU for 500K updates.
The overhead comes from two additional state-only functions that Opt-SUNDER trains alongside V: the policy-normalization multiplier λ(s) and the auxiliary state function ed^(s) (Appendix E.1). The optimization remains a single-level minimization with one extra regression per update; it adds no inner loop, adversarial step, or extra pass over the data. In absolute terms, the overhead is about 2.3 minutes of training time and 130 MiB of memory. The table measures the DICE training stage. Policy extraction and inference use the same weighted flow-matching procedure and policy architecture for both methods, so their cost is identical.
The additional coefficient also does not require a larger search. With an equal three-candidate budget and α fixed to 1, Opt-SUNDER still outperforms OptiDICE (83.96 ± 0.49 vs. 77.94 ± 0.73 over 27 tasks; Table 7), and choosing (α,β) from dataset statistics without per-task tuning retains most of the performance (84.01 vs. 85.44; Table 12).
[W2. Practical Error Propagation and Estimation]
Thank you for this suggestion. As in prior DICE methods, our duality results characterize the population problem, and we agree that the paper should make explicit where the neural implementation departs from it and how the resulting errors propagate. We have revised Appendix E accordingly.
1. Finite-data evidence already in the paper. Not all of our results are population-level. Proposition 2 (Proposition 4 in Appendix C.1) is a finite-data statement: the reward of the rarely observed action is estimated from a single noisy sample, and the gap arises from this sampling noise. A joint coefficient weak enough to move visitation toward the poorly covered state also makes the policy trust the noisy estimate, whereas separate coefficients avoid this trade-off. The Fourrooms study likewise uses 1,000 independently sampled finite buffers with estimated transitions, and the advantage persists when the transition estimator is changed (949 vs. 469 and 898 vs. 427 successes; Table 13). All benchmark results are reported with standard errors over eight seeds (five for LfO).
2. Where approximation enters (Appendix E). The algorithm never represents the occupancy explicitly. It minimizes the dual over (V,λ), which is convex in these functions, and the divergences enter only through conjugates f∗ evaluated as sample averages, so no divergence is estimated directly. Three approximations remain. (i) The action expectation inside the outer conjugate, which is specific to the decoupled dual: a single-sample plug-in would be biased by Jensen's inequality, so we estimate it with an auxiliary regression (Eq. 123) and use the chain-rule loss (Eq. 125), whose gradient equals that of the training objective under exact conditional estimates. (ii) Transition expectations, which are exact in our deterministic benchmarks and can be handled by a state–action auxiliary estimator otherwise. (iii) The initial-state term, approximated by a buffer average (Eqs. 119–120). Ent-SUNDER additionally uses a fixed histogram estimate of u/q^B (Eqs. 128–129), and LfO-SUNDER (KL) a pretrained transition discriminator (Eqs. 144–145).