acceptodds
Under review as a conference paper at ICLR 2027

SupportDICE: Replay-Supported Occupancy Optimization for Offline-to-Online Reinforcement Learning

Abstract

Offline-to-online reinforcement learning aims to reuse an offline dataset while adapting as new interaction data arrives. We study this regime from the DICE (distribution-correction estimation) occupancy-optimization perspective. The central difficulty is that the replay buffer keeps changing: it tells us which state-action pairs have been observed and provides samples for training, but raw replay counts need not themselves form a valid discounted occupancy for policy optimization. We propose SupportDICE, which separates these roles by constructing an augmented replay-supported MDP whose exits from the currently covered region are redirected to an absorbing state. The resulting update optimizes valid occupancies on the current replay support, while a full-support anchor lets newly observed, flow-feasible state-action pairs enter future updates. In finite MDPs, the exact population updates have zero duality gap under finite-divergence feasibility; the fixed-support offline phase recovers DICE improvement in the augmented MDP, while the support-expanding online phase satisfies a per-step comparator guarantee on the current replay-supported problem. We also extend the population analysis to dominated standard Borel spaces. A tabular square-grid example shows that support expansion can expose a better feasible occupancy, while replay-centered DICE remains biased toward the old replay distribution. On D4RL benchmarks, SupportDICE achieves the highest average final scores on Adroit and Kitchen among the evaluated methods, while remaining competitive on AntMaze.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, offline to online

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_THagOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the evolving replay support in O2O RL from the perspective of DICE-based occupancy optimization and proposes SupportDICE. The core idea is to construct an augmented MDP with an absorbing state over the current replay support, while using a full-support anchor to allow newly collected state-action pairs to enter subsequent occupancy optimization. The authors provide population-level theoretical analyses for both finite MDPs and continuous spaces, and evaluate the method on D4RL AntMaze, Adroit, and Kitchen.

Critical strengths

The paper clearly distinguishes between replay as a sampling source and occupancy that actually satisfies Bellman flow, and provides a unified occupancy-space formulation for the dynamically expanding replay support in O2O RL.

Critical weaknesses
  • The theory is centered on a replay-supported augmented MDP, where transitions leaving the current support are mapped to an absorbing state. However, Appendix C.5 explicitly states that the actual implementation uses sampled DICE backups, processed replay weights, and parameterized policy learning, while the absorbing state only appears in the population formulation and is not used in the numerical update. Therefore, Theorems 1–3 do not directly explain or guarantee the empirical performance on D4RL.

  • The experimental baselines are not sufficiently up to date, and some of the most relevant methods are missing. The paper already cites Uni-RL 2026 and directly adopts the FQL 2025 policy architecture in the implementation, but the main experiments only compare against Cal-QL, AWAC, IQL, CQL, SPOT, RLPD, and PEX. Uni-RL is not included, and the most important matched FQL baseline is also missing.

  • AntMaze and Adroit use 1M online transitions, while Kitchen uses 1.25M. This is somewhat inconsistent with the paper's emphasis on limited online interaction.

  • The paper lacks corresponding ablations. For example, the DICE λ\lambda and backward-gradient scale are set differently across tasks, while the policy guidance coefficient α\alpha ranges from 10 to 10000. Kitchen further introduces a bounded Q head, target clipping, twin-Q minimum, and guidance scaling. These choices may have a substantial effect on the final results, but the paper does not explain the tuning rules or isolate their individual contributions.

Questions for authors

See Weaknesses.

1 comment

Thank you again for your time in reviewing our paper. As the discussion period draws to a close, we would be grateful if you could take a look at our responses to your comments and share any follow-up feedback.

We would be happy to clarify any remaining questions or concerns before the discussion period closes.

Reviewer_QE4BOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes SupportDICE, a replay‑supported augmented MDP that redirects transitions that leave current observed support to an absorbing absorbing terminal state. The authors also suggest an anchored reference measure with full‑support base measure removes divergence barrier caused by zero‑mass reference at newly‑added samples, and theoretical guarantee are further provided. Empirical studies on D4RL demonstrate the superiority of the proposed method.

Critical strengths
  • The studied problem is interesting and the proposed solution is novel. For O2O setting with growing replay support, an absorbing state redirects transitions leaving observed support. An augmented MDP is built in each iteration to define valid occupancy feasible set, turning online sample‑adding process into sequential convex optimization.

  • The anchored reference mechanism is appealing. The mechanism removes f‑divergence divergence barrier on newly‑added state‑action pairs, so Bellman‑flow‑feasible new samples can enter occupancy optimization.

  • The manuscript is well-organized, and the theoretical framework of the proposed method is comprehensive.

Critical weaknesses
  • Although the theoretical results in the manuscripts are appealing. However, using neural‑network practical implementation may introduce function‑approximation error, which lacks theoretical treatment.

  • Hyper‑parameter dependency: anchor mixing coefficient ε\varepsilon, divergence regularization weight η\eta require tuning. Fixed parameters used in the manuscript, and further ablation studies are suggested.

Questions for authors
  • The experiment studies are performed on D4RL dataset. Is there any real-scenario experiment?

  • The augmented‑return mismatch is only a pessimistic upper bound. Is there any approach to empieically measure the practical magnitude across tasks.

  • Can adaptive‑ε\varepsilon scheme be designed to automatically adjust anchor mixing weight according to replay‑support scale and sample count?

1 comment

Thank you again for your time in reviewing our paper. As the discussion period draws to a close, we would be grateful if you could take a look at our responses to your comments and share any follow-up feedback.

We would be happy to clarify any remaining questions or concerns before the discussion period closes.

Reviewer_gpwNOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies offline-to-online (O2O) RL from the DICE / occupancy-optimization perspective. It argues that raw replay frequencies in O2O need not form a valid discounted occupancy, and that a DICE update regularized toward a carried reference cannot place mass on newly observed coordinates ("support trapping"). The proposed method, SupportDICE, has two components:

  1. Replay-supported augmented MDP. Transitions that leave the replay-covered state set are redirected to a zero-reward absorbing state ⊥\perp, giving a Bellman-flow-feasible occupancy set Dkaug\mathcal{D}^{aug}_k on the current support.
  2. Anchored reference. The f-divergence reference is qk=(1−ε)qˉk+εmkq_k = (1-\varepsilon)\bar q_k + \varepsilon m_k, mixing the previous occupancy (zero-extended) with a full-support base measure, so newly covered coordinates have finite divergence.

Theory (finite MDPs): zero duality gap under finite-divergence feasibility (Thm 1), pointwise Fenchel/KKT recovery with a careful treatment of dual attainment and structural zeros, fixed-support monotone improvement (Thm 2), a per-step comparator bound on the current support (Thm 3), an augmentation-mismatch lemma linking augmented and original returns, and a dominated continuous-space extension (App. B).

Validation: an exact 9×9 gridworld in which SupportDICE moves to higher-reward branches as they enter support while replay-centered DICE stays on the low-reward branch; and D4RL O2O experiments (6 AntMaze, 4 Adroit-cloned, 3 Kitchen; 4 seeds) against Cal-QL, AWAC, IQL, CQL, SPOT, RLPD, PEX. The practical algorithm uses an ODICE-style value solver to compute Pearson-recovered replay weights that weight the flow-teacher loss of an FQL-style actor. SupportDICE reports the best mean on Adroit (114.3 vs. 84.6 for RLPD) and Kitchen (80.6 vs. 76.5), and is on par with Cal-QL on AntMaze.

Critical strengths

S1. A careful and correct occupancy-space formalization of the O2O setting. The paper clearly separates two roles of the replay buffer, as a sample source and as a support indicator, and observes that replay frequencies are generally not Bellman-flow feasible. The augmented MDP with an absorbing state gives a well-defined, nonempty feasible set on the current support. The theoretical treatment is more careful than is typical for DICE papers: dual attainment versus mere zero gap is distinguished, structural-zero/inactive coordinates are handled explicitly (including when shifted-KL fails to admit a finite dual minimizer and why Pearson attainment is automatic), and the continuous-space extension states its domination assumptions explicitly. This matters because it gives a principled vocabulary for a question (how should the DICE feasible set and reference evolve as replay grows?) that prior O2O work treats only heuristically.

S2. Strong empirical results on hard manipulation tasks. On Adroit-cloned, SupportDICE improves the mean by roughly 30 points over the next-best method, and on relocate it reaches ~74 while all baselines stay near zero (Fig. 2, Table 3). Kitchen partial also shows a large gain. If these gains can be attributed to the proposed mechanism, they would be practically significant.

Critical weaknesses

W1. The theory is only loosely connected to the evaluated algorithm. The central construct, the augmented MDP in which support exits are redirected to an absorbing state ⊥\perp, has no counterpart in the implementation. App. C.5 states that "the absorbing state enters the population formulation; the numerical update uses the replay losses specified above," and the only terminal handling in the DICE residual is the dataset's own terminal mask. In continuous spaces, the replay-covered set SkenvS^{env}_k, and hence "support expansion" or "leaving support," is never operationalized; App. B.6 says only that these sets are "modeled as measurable regions." The remaining components are formally permitted by the theory, but they depart from the configuration the paper uses to motivate and validate the method:

  • The gridworld and the finite theory use a base measure that is uniform over supported coordinates, mk=Unif(Ωkaug)m_k = \mathrm{Unif}(\Omega^{aug}_k), whereas the implementation uses uniform sampling over replay rows (App. C.3), i.e., the empirical replay distribution, which is offline-dominated early in fine-tuning. App. A.3 allows mk=hkm_k = h_k, so this is not a formal violation. However, it means the anchor component reintroduces the replay distribution, and the gridworld, which argues against replay-centered references, does not test the configuration actually used.
  • The carried reference qˉk\bar q_k is a processed weight vector: a min over DICE value estimates, rescaled and capped at 3 (Eqs. 122–123). The theory allows an arbitrary qˉk\bar q_k, but this object is neither flow-feasible nor the DICE optimizer. The "carry the previous occupancy" interpretation, and Theorem 2 (which requires qk=dk∈Dkaugq_k = d_k \in \mathcal{D}^{aug}_k), therefore do not apply to the practical algorithm.
  • Consistent with zero-extension, new rows receive zero teacher weight until the next refresh (up to 5,000 steps) and then a reference ratio of ε=0.3\varepsilon = 0.3, below the average of 1. Since the critic and the student are trained on uniformly sampled replay, new data mainly influence the policy through mechanisms outside the proposed reference construction.

The guarantees themselves are standard. Theorems 2 and 3 follow from a direct optimality comparison; Theorem 2 by design covers only the fixed-support phase (d≪dkd \ll d_k); and the per-step gap in Theorem 3 does not vanish. It is 1ηlog⁡(∣Ωkaug∣/ε)\frac{1}{\eta}\log(|\Omega^{aug}_k|/\varepsilon) for shifted KL and scales like ∣Ωkaug∣/ε|\Omega^{aug}_k|/\varepsilon for the Pearson generator used in the experiments. Taken together, the theory does not explain the reported D4RL gains.

W2. The empirical gains cannot be attributed to the proposed mechanism.

  • The actor is FQL (Park et al., 2025). The DICE weights enter only the flow-teacher regression and reach the deployed one-step student only indirectly through distillation, alongside Q-guidance. FQL itself, which reports O2O results, is not a baseline. There is no FQL-with-uniform-weights ablation, no plain-DICE (e.g., ODICE/OptiDICE) O2O baseline on D4RL, and no ablation over ε\varepsilon (ε=0\varepsilon = 0, ε=1\varepsilon = 1, or no reweighting). The paper contains no ablations at all.
  • All baselines date from 2020–2023. Methods the paper itself cites are not compared, including WSRL (whose Kitchen reward protocol is adopted), Uni-O4, Uni-RL, ODICE, and Diffusion-DICE.
  • There is no indication of comparable tuning effort. Baseline numbers come from CORL fine-tuning runs or released implementations, while SupportDICE uses per-task-group actor coefficients (α=1000\alpha = 1000 for pen/door/hammer vs. 1000010000 for relocate), per-task DICE λ\lambda and backward-gradient scales, and Kitchen-specific stabilization (bounded sigmoid Q-head, TD-target clipping, guidance divided by 400) that is not reported for the Q-learning baselines scoring near zero on Kitchen.
  • Only 4 seeds are used. AntMaze has little discriminative power (5 of 8 methods average above 90), and no OGBench results are reported, although OGBench is the primary benchmark for the FQL line of work.

The gridworld does not close this gap. It lacks an ε=0\varepsilon = 0 control (which would demonstrate support trapping) and an ε=1\varepsilon = 1 / uniform-reference control (which would test whether carrying the previous occupancy matters). This is important because at k=1k = 1 the carried SupportDICE occupancy places no mass on the middle branch, so mass there comes entirely from the εmj\varepsilon m_j term. In addition, the replay weights (280 on low-branch forward edges, 0.04 on middle-branch forward edges, 80 on noisy high-branch edges) make the better branches rare under replay by construction, which largely predetermines the replay-centered outcome.

Questions for authors
  1. Ablations within the proposed pipeline. Please report D4RL O2O results, under identical actor/critic hyperparameters, for: (a) FQL with uniform teacher weights (no DICE reweighting); (b) SupportDICE with ε=0\varepsilon = 0, under which new rows keep zero reference mass, i.e., the support-trapping regime the paper motivates against; and (c) SupportDICE with ε=1\varepsilon = 1, under which the reference is the replay sampling distribution, i.e., replay-centered DICE weighting. If (a) or (c) matches SupportDICE on Adroit and Kitchen, the gains would not be attributable to the proposed reference mechanism. This is the single most important question for my assessment.

  2. Standalone baselines. How do FQL (as published, with its own O2O protocol) and an O2O variant of ODICE or OptiDICE perform under your evaluation protocol? A comparison with at least one recent O2O method the paper cites (e.g., WSRL, whose Kitchen protocol is adopted) would also help place the results.

  3. Base measure in practice. The finite theory and the gridworld use a base measure uniform over supported coordinates, while the implementation samples uniformly over replay rows (App. C.3). App. A.3 permits mk=hkm_k = h_k, but then the anchor component reintroduces the replay distribution, which is the reference the gridworld argues against. How should the gridworld be read as evidence for the practical algorithm? Since "uniform over supported coordinates" is not defined in continuous spaces, did you try a density-corrected or deduplicated base measure, and does it change the results?

  4. Role of the augmented MDP in the implementation. How, concretely, are "support expansion" and "leaving support" realized in continuous control? App. C.5 indicates that ⊥\perp is not instantiated, and the only terminal handling appears to be the dataset terminal mask. Which component of the D4RL algorithm is specific to the augmented MDP, as opposed to generic advantage-weighted reweighting with a mixed reference?

  5. Tuning protocol and baseline provenance. How were SupportDICE's per-task hyperparameters (actor coefficient α\alpha, DICE λ\lambda, backward-gradient scale, Kitchen stabilizations) selected, and were they selected on the same seeds and final evaluations reported in Table 1? Were baselines given a comparable tuning budget? Were the Kitchen stabilizations (bounded Q-head, TD-target clipping) tried for Q-learning baselines such as Cal-QL and CQL, which score at or near zero on Kitchen? Where do the PEX numbers come from?

  6. Gridworld controls. Please report ε=0\varepsilon = 0 and ε=1\varepsilon = 1 in the gridworld. At k=1k = 1 the carried SupportDICE occupancy places no mass on the middle branch, so mass there comes entirely from the εmj\varepsilon m_j term; an ε=1\varepsilon = 1 control would show whether carrying the previous occupancy matters at all. Results under replay weights in which the better branches are less rare would also show how general the observed separation is.

  7. Absorbing-state reward. The theory assigns zero reward at ⊥\perp, while the experiments use nonpositive reward conventions (AntMaze −1/0 for the policy and −3/0 for DICE; Kitchen [−4,0][-4, 0]). Under these conventions the augmented MDP would make exiting support attractive. Which reward convention should the theory be read under, and would a pessimistic absorbing reward (e.g., per-step rmin⁡r_{\min} at ⊥\perp) change any of the stated results?

  • Report OGBench results, which are the primary benchmark for FQL-based methods.
  • Use more than 4 seeds and report confidence intervals.
  • State explicitly in the main text which theoretical objects are not instantiated in the practical algorithm.
2 comments

Thank you again for your time in reviewing our paper. As the discussion period draws to a close, we would be grateful if you could take a look at our responses to your comments and share any follow-up feedback.

We would be happy to clarify any remaining questions or concerns before the discussion period closes.

Thank you for the detailed responses and additional experiments. The new FQL baseline, reference ablations, and gridworld controls substantially improve the empirical picture. In particular, the no-anchor results support the concern that assigning zero reference mass to newly collected data can impair online adaptation. However, my main concern about attributing the reported gains to the proposed mechanism remains unresolved.

  1. The FQL comparison does not replace the requested controlled ablation. FQL matches SupportDICE on AntMaze and Adroit, suggesting that the previously highlighted Adroit advantage over the original baselines does not establish a benefit from the proposed weighting mechanism. The Kitchen improvement remains interesting, but the response evaluates official FQL with its own settings, rather than the identical SupportDICE pipeline with uniform teacher weights. Please clarify whether the Kitchen critic architecture, target clipping, guidance scaling, and all other training settings are identical. Otherwise, the source of this improvement remains uncertain.
  2. The results do not establish a benefit from carrying the previous occupancy. The full-support-base reference performs comparably to the mixed reference across the three task groups, and also performs well in the gridworld. I understand that this endpoint belongs to the proposed family and is covered by the stated theory. Nevertheless, membership in the family does not demonstrate the usefulness of the carried-reference component. The evidence currently supports ensuring positive reference mass on new data more directly than it supports the proposed history-dependent construction. The paper should distinguish these claims and adjust its motivation accordingly.
  3. The population-to-implementation correspondence needs clearer limits. The explanation of stored successors and terminal masks is helpful. However, allowing an arbitrary processed reference in Theorems 1 and 3 does not by itself establish guarantees for the practical combination of approximate value fitting, weight clipping, and actor distillation. Please state explicitly which guarantees apply to the population optimization problem and which, if any, carry over to the implemented algorithm.

Overall, the response addresses several missing controls, but the central attribution question remains open.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.