- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the evolving replay support in O2O RL from the perspective of DICE-based occupancy optimization and proposes SupportDICE. The core idea is to construct an augmented MDP with an absorbing state over the current replay support, while using a full-support anchor to allow newly collected state-action pairs to enter subsequent occupancy optimization. The authors provide population-level theoretical analyses for both finite MDPs and continuous spaces, and evaluate the method on D4RL AntMaze, Adroit, and Kitchen.
Critical strengths
The paper clearly distinguishes between replay as a sampling source and occupancy that actually satisfies Bellman flow, and provides a unified occupancy-space formulation for the dynamically expanding replay support in O2O RL.
Critical weaknesses
-
The theory is centered on a replay-supported augmented MDP, where transitions leaving the current support are mapped to an absorbing state. However, Appendix C.5 explicitly states that the actual implementation uses sampled DICE backups, processed replay weights, and parameterized policy learning, while the absorbing state only appears in the population formulation and is not used in the numerical update. Therefore, Theorems 1–3 do not directly explain or guarantee the empirical performance on D4RL.
-
The experimental baselines are not sufficiently up to date, and some of the most relevant methods are missing. The paper already cites Uni-RL 2026 and directly adopts the FQL 2025 policy architecture in the implementation, but the main experiments only compare against Cal-QL, AWAC, IQL, CQL, SPOT, RLPD, and PEX. Uni-RL is not included, and the most important matched FQL baseline is also missing.
-
AntMaze and Adroit use 1M online transitions, while Kitchen uses 1.25M. This is somewhat inconsistent with the paper's emphasis on limited online interaction.
-
The paper lacks corresponding ablations. For example, the DICE and backward-gradient scale are set differently across tasks, while the policy guidance coefficient ranges from 10 to 10000. Kitchen further introduces a bounded Q head, target clipping, twin-Q minimum, and guidance scaling. These choices may have a substantial effect on the final results, but the paper does not explain the tuning rules or isolate their individual contributions.
Questions for authors
See Weaknesses.
Thank you again for your time in reviewing our paper. As the discussion period draws to a close, we would be grateful if you could take a look at our responses to your comments and share any follow-up feedback.
We would be happy to clarify any remaining questions or concerns before the discussion period closes.