- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DRIFT, an offline-to-online reinforcement learning method for discrete action spaces using continuous-time Markov chain (CTMC) policies. The proposed method trains the CTMC generator through discrete flow matching and constructs an advantage-weighted target policy. Furthermore, they introduce path-space KL regularization, which constrains the CTMC trajectory distribution rather than only the final action distribution, which is a key component of this work. In addition, a candidate-set approximation based on reference-policy rollouts and uniform sampling is proposed to reduce computational costs in large discrete action spaces.
Critical strengths
S1 (Important problem setting). The paper explores offline-to-online RL with discrete flow matching policies, a relatively unexplored research direction. It extends pretrained discrete flow policies to online fine-tuning, particularly for large discrete action spaces.
S2 (Path-space KL regularization). The proposed path-space KL regularization provides an approach to preventing abrupt policy changes during online fine-tuning by constraining deviations from the reference CTMC sampling process. Unlike conventional final-flow-step-action KL regularization, it considers the full CTMC trajectory distribution, including transition rates and holding times. Although this may introduce additional computational overhead.
S3 (Theoretical analysis of candidate-set approximation). The paper provides a theoretical analysis of the candidate-set approximation, establishing bounds on the target-policy approximation error and the corresponding CTMC generator error. These results offer useful theoretical insights into the trade-off between candidate coverage and approximation accuracy in large discrete action spaces.
S4 (Empirical evaluation). The paper evaluates the proposed method on macro-action MinAtar and Jericho text games, demonstrating its applicability to combinatorial and text-based discrete action spaces. The comparisons with existing methods, including RL-D^2 and QDFM.
Critical weaknesses
W1 (Insufficient justification for using flow-based policies). In Section 2, the authors state that existing discrete-action RL methods, including PPO, "do not model multimodal action distributions." However, this statement is inaccurate, as standard categorical policies can already represent multimodal distributions over discrete actions. Therefore, the comparison with DQN in Figure 1(b) does not sufficiently demonstrate the benefits of flow-based policies over other stochastic policy parameterizations.
W2 (Limited evidence for effective exploration through candidate-set construction). The proposed method constructs candidate sets using reference-policy samples and uniform sampling, with the latter intended to discover better actions beyond the pretrained policy. However, uniform sampling can be inefficient in large discrete action spaces, where high-value actions may occupy only a small fraction of the action space (cf. emergence of MCMC). Although Table 5 evaluates reference-free candidates, it does not isolate the contribution of uniform sampling or demonstrate that the combined strategy improves exploration over reference-only sampling. Additional analyses would help establish the effectiveness of the proposed candidate construction for online exploration.
W3 (Unclear effects of periodic reference policy updates). The proposed method periodically replaces the reference generator with the current CTMC policy . However, the paper does not investigate how this design affects online policy improvement or the preservation of pretrained behavior. A comparison across four settings, DRIFT with a fixed reference, periodic updates, and updates at every step, as well as RL-D^2 with periodic reference () updates, would help clarify the contribution of the proposed reference update strategy relative to the proposed components.
W4 (Insufficient hyperparameter details and sensitivity analysis). Several important hyperparameters are insufficiently specified or analyzed, particularly the reference refresh interval and the advantage clipping threshold . The choice of is especially important because clipping could suppress differences between action advantages and prevent the target policy from adequately reflecting the estimated advantages. Providing the exact settings and sensitivity analyses for these hyperparameters would improve reproducibility and help assess the robustness of DRIFT.
Questions for authors
Could the authors provide additional explanations or experimental results addressing the concerns raised above?
6 comments
Response to Reviewer 6Rxq (2/2): W3, W4 (Reference updates and hyperparameter sensitivity)
"W3 (Unclear effects of periodic reference policy updates). proposed method periodically replaces reference generator with current CTMC policy . However, paper does not investigate how this design affects online policy improvement or preservation of pretrained behavior. comparison across four settings, DRIFT with fixed reference, periodic updates, updates at every step, as well as RL-D^2 with periodic reference () updates, would help clarify contribution of proposed reference update strategy relative to proposed components."
We investigated the effect of periodic reference policy updates on Freeway (, five seeds), comparing the four settings: fixed, periodic (, default), updates at every step, and RL-D²-O2O with periodic reference updates.
| Configuration | Final return ↑ | KL to original policy ↓ |
|---|---|---|
| DRIFT (fixed) | 34.5±1.6 | 0.19 |
| DRIFT (periodic, default) | 38.1±0.5 | 0.38 |
| DRIFT (every update) | 26.0±1.1 | 0.74 |
| RL-D²-O2O (original) | 32.8±2.3 | 0.52 |
| RL-D²-O2O (periodic) | 35.1±1.8 | 0.49 |
Periodic refreshing achieves highest return among tested configurations. A fixed reference maintains greater similarity to pretrained policy but limits adaptation, whereas updating at every step allows greater drift and reduces performance. Periodic updating balances these effects by allowing the reference to track policy improvement without changing after every update.
The RL-D² comparison further indicates that reference refreshing can benefit other methods, while DRIFT achieves higher returns in this setting. We have clarified the reference update strategy in Section 4 under Overview and added the comparison in Appendix D.5 under Reference updates.
"W4 (Insufficient hyperparameter details and sensitivity analysis). hyperparameters insufficiently specified or analyzed, particularly reference refresh interval K and advantage clipping threshold c. choice of c is important because clipping could suppress differences between action advantages and prevent target policy from adequately reflecting estimated advantages. Providing exact settings and sensitivity analyses for these hyperparameters would improve reproducibility and help assess the robustness of DRIFT."
Our macro-MinAtar experiments use reference-refresh interval actor updates and advantage clipping threshold . Advantages are normalized within each candidate set before clipping to .
Reference-refresh sensitivity. As shown in W3, our comparison of fixed, periodic, and every-update references examines the trade-off between online improvement and preservation of pretrained behavior.
Advantage-clipping sensitivity. We additionally evaluated different clipping thresholds on Freeway (, five seeds), including no clipping.
| Clipping threshold | Final return ↑ | Advantages clipped |
|---|---|---|
| 33.4±1.3 | 30.5% | |
| 35.5±0.9 | 6.7% | |
| (default) | 38.1±0.5 | 1.9% |
| 37.1±0.6 | 0.6% | |
| No clipping | 36.5±1.3 | 0% |
Aggressive clipping () affects 30.5% of advantages and gives the lowest return, consistent with suppressing useful advantage differences. The default clips only 1.9% and achieves the highest mean return. Performance remains reasonably strong with weaker or no clipping, suggesting that DRIFT is robust and does not depend on a narrowly chosen threshold.
We have specified the clipping threshold in Section 4.3 and clarified the reference refresh interval in Section 4 under Overview of the revised manuscript. The complete reference update and advantage clipping sensitivity results are provided in Appendix D.5 under Reference updates and Advantage clipping sensitivity.
We hope this addresses the reviewer's concerns and are happy to answer further questions.
Dear Reviewer,
As the discussion period is nearing its end, we hope our responses and additional experimental results have addressed your concerns. We would be happy to provide any further clarification if needed and would appreciate your feedback.
Response to Reviewer 6Rxq (1/2): W1, W2 (CTMC policies and candidate exploration)
We sincerely thank the reviewer for their time and helpful feedback, and for recognizing the importance of our problem setting, path-space KL regularization, theoretical analysis, and diverse experiments. We address your concerns below.
Since categorical policies like PPO can represent multimodal distributions, our motivation for CTMC policies is not only representational capacity, but their explicit stochastic generation process, which allows us to regularize the entire sampling trajectory through path-space KL during online adaptation. Our additional experiments demonstrate this distinction's practical benefits.
We compared DRIFT with path-space KL, CTMC and autoregressive categorical actors under terminal-policy KL, and offline-initialized PPO on Freeway (K=4, five seeds).
The categorical and CTMC actors achieve comparable returns (35.2 vs. 34.9) and action diversity (27 vs. 26) under terminal KL, suggesting CTMC parameterization alone does not explain DRIFT's improvement. However, DRIFT with path-space KL achieves higher returns (38.1) and maintains greater final policy entropy (0.35 vs. 0.23-0.24) and action diversity (37 vs. 26-27 distinct actions).
These results support our motivation for using CTMC policies. Their explicit sampling trajectories allow path-space regularization to preserve useful generation dynamics during online adaptation, providing benefits beyond conventional terminal-policy regularization.
We have corrected the statement about categorical policy expressivity in Section 2, summarized the controlled actor comparison in Section 5 under Ablations, and provided the complete results in Appendix D.5 under CTMC and categorical policy comparison in the revised manuscript. Figure 1(b) illustrates the observed comparison with DQN rather than an inherent expressivity advantage over all categorical policies.
We conducted additional candidate-composition ablations and provenance diagnostics on Freeway at K=4 and K=8, with B=64 and five seeds.
Candidate-composition ablation. We compared DRIFT's mixed candidate construction against reference-only sampling without uniform filling.
The mixed strategy achieves slightly higher mean returns at K=4, while reference-only achieves nearly identical performance at K=8. Both substantially outperform reference-free baseline.
Candidate-provenance diagnostics. We measured uniform candidates' contribution to the advantage-weighted target.
Although uniform filling occupies 90% of candidate slots at K=4, uniform-exclusive actions receive only 1.7% of the candidate-restricted target mass (0.04% at K=8). Thus, a small number of reference/replay-supported actions dominate the learning target, even when most slots are filled uniformly. This helps explain why DRIFT's candidates capture 26% of the full-action target mass, versus 4.9% expected from 64 uniformly random actions (Appendix D.4). Uniform filling may still improve diversity when reference rollouts produce repeated actions.
The reference-free ablation (11.1 vs. 38.1 on Freeway) also removes reference-frequency weighting, whereas our new reference-only comparison more directly isolates the contribution of uniform filling. These results highlight the importance of reference-guided candidate selection and motivate more targeted proposals, such as local perturbations of reference actions to further improve scalability.
We clarified candidate construction in Section 4.2, summarized the reference-only comparison in Section 5, and added composition and provenance diagnostics in Appendix D.5.