acceptodds
Under review as a conference paper at ICLR 2027

Discrete Flow Matching for Offline-to-Online Reinforcement Learning

Abstract

Many reinforcement learning (RL) tasks have discrete action spaces, but most generative policy methods based on diffusion and flow matching are designed for continuous control. Meanwhile, generative policies usually rely heavily on offline datasets and offline-to online RL is itself challenging, as the policy must improve from new interaction without losing useful behavior learned from static data. To address those challenges, we introduce DRIFT, an online fine-tuning method that updates an offline pretrained continuous time Markov chain (CTMC) policy with an advantage-weighted discrete flow matching loss. To preserve useful pretrained knowledge, we add a path-space penalty that regularizes the full CTMC trajectory distribution, rather than only the final action distribution. For large discrete action spaces, we introduce a candidate-set approximation that updates the actor over a small subset of actions sampled from reference-policy rollouts and uniform exploration. Our theoretical analysis shows that focusing on a small set of high probability actions can reduce computation while keeping the approximation error small, making our method practical for large action spaces. We evaluate on macro-action MinAtar games where offline-pretrained DRIFT outperforms its online-only counterpart across all five games and achieves the highest mean return on three and on oracle free Jericho, DRIFT achieves a mean normalized score of 7.3% with a lightweight GRU encoder, outperforming the evaluated pretrained language-model baselines. Ablations demonstrate the importance of appropriately weighted path-space regularization and reference-guided candidate construction.

Primary area:
reinforcement learning
Keywords:
offline-to-online reinforcement learning, discrete flow matching, discrete action spaces, generative policies, continuous-time Markov chains

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6RxqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DRIFT, an offline-to-online reinforcement learning method for discrete action spaces using continuous-time Markov chain (CTMC) policies. The proposed method trains the CTMC generator through discrete flow matching and constructs an advantage-weighted target policy. Furthermore, they introduce path-space KL regularization, which constrains the CTMC trajectory distribution rather than only the final action distribution, which is a key component of this work. In addition, a candidate-set approximation based on reference-policy rollouts and uniform sampling is proposed to reduce computational costs in large discrete action spaces.

Critical strengths

S1 (Important problem setting). The paper explores offline-to-online RL with discrete flow matching policies, a relatively unexplored research direction. It extends pretrained discrete flow policies to online fine-tuning, particularly for large discrete action spaces.

S2 (Path-space KL regularization). The proposed path-space KL regularization provides an approach to preventing abrupt policy changes during online fine-tuning by constraining deviations from the reference CTMC sampling process. Unlike conventional final-flow-step-action KL regularization, it considers the full CTMC trajectory distribution, including transition rates and holding times. Although this may introduce additional computational overhead.

S3 (Theoretical analysis of candidate-set approximation). The paper provides a theoretical analysis of the candidate-set approximation, establishing bounds on the target-policy approximation error and the corresponding CTMC generator error. These results offer useful theoretical insights into the trade-off between candidate coverage and approximation accuracy in large discrete action spaces.

S4 (Empirical evaluation). The paper evaluates the proposed method on macro-action MinAtar and Jericho text games, demonstrating its applicability to combinatorial and text-based discrete action spaces. The comparisons with existing methods, including RL-D^2 and QDFM.

Critical weaknesses

W1 (Insufficient justification for using flow-based policies). In Section 2, the authors state that existing discrete-action RL methods, including PPO, "do not model multimodal action distributions." However, this statement is inaccurate, as standard categorical policies can already represent multimodal distributions over discrete actions. Therefore, the comparison with DQN in Figure 1(b) does not sufficiently demonstrate the benefits of flow-based policies over other stochastic policy parameterizations.

W2 (Limited evidence for effective exploration through candidate-set construction). The proposed method constructs candidate sets using reference-policy samples and uniform sampling, with the latter intended to discover better actions beyond the pretrained policy. However, uniform sampling can be inefficient in large discrete action spaces, where high-value actions may occupy only a small fraction of the action space (cf. emergence of MCMC). Although Table 5 evaluates reference-free candidates, it does not isolate the contribution of uniform sampling or demonstrate that the combined strategy improves exploration over reference-only sampling. Additional analyses would help establish the effectiveness of the proposed candidate construction for online exploration.

W3 (Unclear effects of periodic reference policy updates). The proposed method periodically replaces the reference generator urefu_{ref} with the current CTMC policy uθu_{\theta}. However, the paper does not investigate how this design affects online policy improvement or the preservation of pretrained behavior. A comparison across four settings, DRIFT with a fixed reference, periodic updates, and updates at every step, as well as RL-D^2 with periodic reference (πk\pi_k) updates, would help clarify the contribution of the proposed reference update strategy relative to the proposed components.

W4 (Insufficient hyperparameter details and sensitivity analysis). Several important hyperparameters are insufficiently specified or analyzed, particularly the reference refresh interval KK and the advantage clipping threshold cc. The choice of cc is especially important because clipping could suppress differences between action advantages and prevent the target policy π~\tilde{\pi} from adequately reflecting the estimated advantages. Providing the exact settings and sensitivity analyses for these hyperparameters would improve reproducibility and help assess the robustness of DRIFT.

Questions for authors

Could the authors provide additional explanations or experimental results addressing the concerns raised above?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 6Rxq (1/2): W1, W2 (CTMC policies and candidate exploration)

We sincerely thank the reviewer for their time and helpful feedback, and for recognizing the importance of our problem setting, path-space KL regularization, theoretical analysis, and diverse experiments. We address your concerns below.

"W1 (Insufficient justification for using flow-based policies) standard categorical policies can represent multimodal distributions over discrete actions, comparison with DQN in Figure 1(b) does not sufficiently demonstrate benefits of flow-based policies over other stochastic policy parameterizations."

Since categorical policies like PPO can represent multimodal distributions, our motivation for CTMC policies is not only representational capacity, but their explicit stochastic generation process, which allows us to regularize the entire sampling trajectory through path-space KL during online adaptation. Our additional experiments demonstrate this distinction's practical benefits.

We compared DRIFT with path-space KL, CTMC and autoregressive categorical actors under terminal-policy KL, and offline-initialized PPO on Freeway (K=4, five seeds).

MethodReturn ↑Norm. entropy ↑Distinct actions ↑
DRIFT (path KL)38.1±0.50.3537
DRIFT (terminal KL)34.9±1.30.2326
Categorical (terminal KL)35.2±1.50.2427
PPO-Macro-O2O31.7±1.70.1921

The categorical and CTMC actors achieve comparable returns (35.2 vs. 34.9) and action diversity (27 vs. 26) under terminal KL, suggesting CTMC parameterization alone does not explain DRIFT's improvement. However, DRIFT with path-space KL achieves higher returns (38.1) and maintains greater final policy entropy (0.35 vs. 0.23-0.24) and action diversity (37 vs. 26-27 distinct actions).

These results support our motivation for using CTMC policies. Their explicit sampling trajectories allow path-space regularization to preserve useful generation dynamics during online adaptation, providing benefits beyond conventional terminal-policy regularization.

We have corrected the statement about categorical policy expressivity in Section 2, summarized the controlled actor comparison in Section 5 under Ablations, and provided the complete results in Appendix D.5 under CTMC and categorical policy comparison in the revised manuscript. Figure 1(b) illustrates the observed comparison with DQN rather than an inherent expressivity advantage over all categorical policies.

"W2 (Limited evidence for effective exploration through candidate-set construction) uniform sampling can be inefficient in large discrete action spaces, where high-value actions may occupy only small fraction of action space. Although Table 5 evaluates reference-free candidates, it does not isolate contribution of uniform sampling or demonstrate combined strategy improves exploration. Additional analyses would help establish effectiveness of proposed candidate construction for online exploration."

We conducted additional candidate-composition ablations and provenance diagnostics on Freeway at K=4K=4 and K=8K=8, with B=64B=64 and five seeds.

Candidate-composition ablation. We compared DRIFT's mixed candidate construction against reference-only sampling without uniform filling.

ConfigurationK=4K=8
DRIFT (mixed)38.1±0.526.0±0.8
Reference-only37.2±0.726.1±0.7
Reference-free11.1±1.63.6±0.8

The mixed strategy achieves slightly higher mean returns at K=4K=4, while reference-only achieves nearly identical performance at K=8K=8. Both substantially outperform reference-free baseline.

Candidate-provenance diagnostics. We measured uniform candidates' contribution to the advantage-weighted target.

Diagnostic (Freeway, B=64)K=4K=8
Candidate slots filled uniformly90%83%
Uniform-exclusive candidate has highest advantage3.3%0.05%
Restricted target mass on uniform-exclusive actions1.7%0.04%

Although uniform filling occupies 90% of candidate slots at K=4K=4, uniform-exclusive actions receive only 1.7% of the candidate-restricted target mass (0.04% at K=8K=8). Thus, a small number of reference/replay-supported actions dominate the learning target, even when most slots are filled uniformly. This helps explain why DRIFT's candidates capture 26% of the full-action target mass, versus 4.9% expected from 64 uniformly random actions (Appendix D.4). Uniform filling may still improve diversity when reference rollouts produce repeated actions.

The reference-free ablation (11.1 vs. 38.1 on Freeway) also removes reference-frequency weighting, whereas our new reference-only comparison more directly isolates the contribution of uniform filling. These results highlight the importance of reference-guided candidate selection and motivate more targeted proposals, such as local perturbations of reference actions to further improve scalability.

We clarified candidate construction in Section 4.2, summarized the reference-only comparison in Section 5, and added composition and provenance diagnostics in Appendix D.5.

AuthorsOpenReview ↗

Response to Reviewer 6Rxq (2/2): W3, W4 (Reference updates and hyperparameter sensitivity)

"W3 (Unclear effects of periodic reference policy updates). proposed method periodically replaces reference generator urefu_{\mathrm{ref}} with current CTMC policy uθu_\theta. However, paper does not investigate how this design affects online policy improvement or preservation of pretrained behavior. comparison across four settings, DRIFT with fixed reference, periodic updates, updates at every step, as well as RL-D^2 with periodic reference (πk\pi_k) updates, would help clarify contribution of proposed reference update strategy relative to proposed components."

We investigated the effect of periodic reference policy updates on Freeway (K=4K=4, five seeds), comparing the four settings: fixed, periodic (Kref=500K_{\mathrm{ref}}=500, default), updates at every step, and RL-D²-O2O with periodic reference updates.

ConfigurationFinal return ↑KL to original policy ↓
DRIFT (fixed)34.5±1.60.19
DRIFT (periodic, default)38.1±0.50.38
DRIFT (every update)26.0±1.10.74
RL-D²-O2O (original)32.8±2.30.52
RL-D²-O2O (periodic)35.1±1.80.49

Periodic refreshing achieves highest return among tested configurations. A fixed reference maintains greater similarity to pretrained policy but limits adaptation, whereas updating at every step allows greater drift and reduces performance. Periodic updating balances these effects by allowing the reference to track policy improvement without changing after every update.

The RL-D² comparison further indicates that reference refreshing can benefit other methods, while DRIFT achieves higher returns in this setting. We have clarified the reference update strategy in Section 4 under Overview and added the comparison in Appendix D.5 under Reference updates.

"W4 (Insufficient hyperparameter details and sensitivity analysis). hyperparameters insufficiently specified or analyzed, particularly reference refresh interval K and advantage clipping threshold c. choice of c is important because clipping could suppress differences between action advantages and prevent target policy π~\tilde{\pi} from adequately reflecting estimated advantages. Providing exact settings and sensitivity analyses for these hyperparameters would improve reproducibility and help assess the robustness of DRIFT."

Our macro-MinAtar experiments use reference-refresh interval Kref=500K_{\mathrm{ref}}=500 actor updates and advantage clipping threshold c=3c=3. Advantages are normalized within each candidate set before clipping to [−c,c][-c,c].

Reference-refresh sensitivity. As shown in W3, our comparison of fixed, periodic, and every-update references examines the trade-off between online improvement and preservation of pretrained behavior.

Advantage-clipping sensitivity. We additionally evaluated different clipping thresholds on Freeway (K=4K=4, five seeds), including no clipping.

Clipping thresholdFinal return ↑Advantages clipped
c=1c=133.4±1.330.5%
c=2c=235.5±0.96.7%
c=3c=3 (default)38.1±0.51.9%
c=5c=537.1±0.60.6%
No clipping36.5±1.30%

Aggressive clipping (c=1c=1) affects 30.5% of advantages and gives the lowest return, consistent with suppressing useful advantage differences. The default c=3c=3 clips only 1.9% and achieves the highest mean return. Performance remains reasonably strong with weaker or no clipping, suggesting that DRIFT is robust and does not depend on a narrowly chosen threshold.

We have specified the clipping threshold in Section 4.3 and clarified the reference refresh interval in Section 4 under Overview of the revised manuscript. The complete reference update and advantage clipping sensitivity results are provided in Appendix D.5 under Reference updates and Advantage clipping sensitivity.

We hope this addresses the reviewer's concerns and are happy to answer further questions.

Dear Reviewer,

As the discussion period is nearing its end, we hope our responses and additional experimental results have addressed your concerns. We would be happy to provide any further clarification if needed and would appreciate your feedback.

Reviewer_QcVzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on offline-to-online RL with discrete action spaces and proposes DRIFT, an online fine-tuning framework for pretrained CTMC policies. Unlike conventional policy-level regularization, DRIFT regularizes the entire CTMC path distribution to preserve pretrained behavior. To handle large action spaces, the authors introduce a candidate-set approximation and provide theoretical bounds on how its approximation error propagates from the target policy to the CTMC generator. Experiments on macro-action MinAtar and Jericho text games demonstrate the effectiveness of DRIFT, achieving the best performance on 3 out of 5 MinAtar games and an average normalized score of 7.3% on Jericho, compared to 3.6% for the best evaluated oracle-free baseline.

Critical strengths
  1. The paper combines offline initialization, advantage-weighted CTMC updates, and candidate-set approximation into a unified framework, achieving competitive results across multiple benchmarks. I think this demonstrates the practical value of the proposed framework for offline-to-online RL with large discrete action spaces.
  2. I find the theoretical analysis of candidate-set approximation valuable. In particular, Theorem 4.2 shows that, under certain conditions, the approximation error of the terminal target policy can bound the error of the CTMC target generator. This extends the analysis from policy distributions to CTMC transition rates and provides theoretical support for the proposed approximation.
Critical weaknesses
  1. I have some concerns about the novelty of this paper. From my perspective, the method seems more like a framework that combines several existing techniques. Although the authors provide some ablation studies, I do not think they sufficiently demonstrate the unique advantages of each component over existing alternatives, or whether combining these components brings additional benefits.
  2. Regarding path-space KL regularization, which is one of the main contributions of this paper, similar ideas have already been explored in previous works [1, 2]. I think the authors should more clearly explain their contribution. In addition, the paper does not directly demonstrate the advantages of path-space KL over conventional policy KL. I also have some concerns about its gradient implementation. Appendix C.2 mentions that the sampled CTMC trajectories are treated with stop-gradient. However, according to the score-function identity, under exact CTMC sampling, the expected gradient of this surrogate is zero, which raises questions about whether it can provide an effective KL regularization signal. Although the authors acknowledge that this is not an unbiased gradient estimator, I think further clarification is needed on why this surrogate works.
  3. I also have some questions about the theoretical analysis of candidate-set approximation, especially its practical relevance to large action spaces. The theory shows that high probability coverage can lead to small approximation errors, but in the Freeway experiment, the default candidate set only captures about 26% of the target probability mass. Moreover, the error bound in Theorem 4.2 may become very loose as the action space grows. Therefore, I am not sure to what extent these theoretical results support the method's effectiveness in large action spaces.
Questions for authors
  1. What is the unique advantage of path-space KL over conventional policy-level KL? Conventional policy-level KL only constrains the final action distribution, while path-space KL further constrains the entire CTMC trajectory. However, the paper lacks a direct comparison between the two, so it is unclear whether this additional constraint provides any benefits.

  2. Is the gradient implementation of path-space KL effective? Appendix C.2 mentions that the authors apply stop-gradient to the sampled CTMC trajectories. However, according to the score-function identity, under exact CTMC sampling, the expected gradient of this surrogate is zero. This raises concerns about whether it can provide an effective KL regularization signal, and I would appreciate further clarification from the authors.

  3. How meaningful are the theoretical guarantees of candidate-set approximation in practice? The theoretical analysis suggests that high probability coverage leads to small approximation errors, but the candidate set in the Freeway experiment only captures about 26% of the target probability mass. Therefore, I am not sure to what extent these theoretical results can explain the empirical performance, especially in large action spaces.

7 comments
AuthorsOpenReview ↗

Response to Reviewer QcVz (1/2): Q1, Q2, W2 (Path-space KL benefits and gradient implementation)

We sincerely thank the reviewer for their time and helpful feedback, and for recognizing the practical value of our unified framework, competitive results, and theoretical contributions. We address your concerns below.

Q1 & W2. "unique advantage of path-space KL over conventional policy-level KL? lacks direct comparison between two, so unclear whether additional constraint provides benefits."

Policy-level KL constrains only final action probabilities, while path-space KL regularizes CTMC trajectories, including intermediate transitions. Our intuition was this can preserve pretrained generation dynamics not explicitly constrained by terminal KL, potentially helping retain useful behavior during online adaptation. Our direct comparison below supports this hypothesis showing higher returns, lower trajectory divergence, and greater candidate coverage.

We compare all three variants on Freeway (K=4K=4, five seeds), measuring both KL divergences against periodically refreshed reference.

MethodReturn ↑Terminal KL ↓Path KL ↓Coverage ↑
No KL25.9±0.90.440.710.08
Terminal KL34.9±1.30.150.390.19
Path KL (ours)38.1±0.50.180.210.26

Terminal-policy KL achieves slightly lower terminal divergence, but path-space KL achieves 46.2% lower trajectory divergence, higher returns, and greater coverage, supporting preservation of CTMC sampling dynamics. We added the comparison to Sec 5.1 (Table 2), with further analysis in Appendix D.5 of the revised manuscript.

Q2 & W2. "Is gradient implementation of path-space KL effective? Appendix C.2 mentions authors apply stop-gradient to sampled CTMC trajectories. However, according to the score-function identity, under exact CTMC sampling, expected gradient of this surrogate is zero. This raises concerns about whether it can provide effective KL regularization signal, and I would appreciate further clarification from authors."

We thank the reviewer for identifying the discrepancy. The sampled jump-and-holding estimator does have zero expected gradient under exact CTMC sampling. We identified this limitation during development and replaced it with a compensator-based estimator, used in all reported experiments and included in our submitted code. Appendix C.2 inadvertently had the earlier description. We apologize and have corrected it.

The compensator identity gives an equivalent representation of the exact path-space KL. For a simulated CTMC trajectory X0,…,XM−1X_0,\ldots,X_{M-1}, our implemented estimator is

L^KL(s,θ)=∑m=0M−1Δt∑j≠Xm[uθlog⁡uθuref−uθ+uref].\widehat{\mathcal{L}}_{\mathrm{KL}}(s,\theta) = \sum_{m=0}^{M-1}\Delta t \sum_{j\neq X_m} \left[ u_\theta\log\frac{u_\theta}{u_{\mathrm{ref}}} -u_\theta+u_{\mathrm{ref}} \right].

Rates are evaluated at (Xm→j,tm∣s)(X_m\to j,t_m\mid s) with Δt=1/M\Delta t=1/M. This replaces realized jumps by conditional rate-weighted expectations, gradients flow through uθu_\theta with sampled states detached, yielding nonzero local KL gradients.

We evaluated 20 four-state CTMC configurations against exact KL gradients:

EstimatorCosine ↑Rel. error ↓Norm. ΔKL ↓Grad. var. ↓P[descent] ↑
Exact1.0000%−0.98601.000
SampledN/A100%+0.05531.7910.501
Compensator (ours)0.99718.3%−1.0000.1301.000
Score-function0.99856.9%−0.81935.5530.675

The compensator reduced true path-space KL in all 20 configurations, compared with only 4/20 for the sampled estimator. Its median gradient cosine similarity was 0.9971 with 8.3% relative error and KL descent remained reliable across all tested configurations.

These results support the compensator as a practical regularizer. We clarified its derivation, gradient analysis, unbiased comparison, and numerical validation in Section 4.4 and Appendix C.2.

W2. "Regarding path-space KL, similar ideas explored in [1, 2]. I think authors should more clearly explain their contribution. paper does not directly demonstrate advantages of path-space KL over conventional policy KL. also some concerns about its gradient implementation. Although authors acknowledge this is not an unbiased estimator, I think further clarification is needed."

Q1, Q2 address the KL comparison and gradient concern. We clarified distinctions from related CTMC methods in Section 2 and welcome the missing references [1, 2].

Our contribution is not path-space KL alone: candidate restriction changes advantage-weighted targets and CTMC transition rates, exploratory actions may have zero reference probability, and intermediate states may lie outside the candidate set. We address these difficulties using smoothed reference weights, candidate-normalized advantages, and a valid independent-coupling target generator. Our estimator provides meaningful rate gradients without differentiating through sampled trajectories. Our joint ablation (W1) shows positive interactions of 10.6 points on Freeway and 13.7 on Space Invaders, demonstrating benefits beyond the individual components.

AuthorsOpenReview ↗

Response to Reviewer QcVz (2/2): Q3, W3, W1 (Theory and novelty clarifications)

Q3 & W3. "How meaningful are theoretical guarantees of candidate-set approximation in practice? theoretical analysis suggests high probability coverage leads to small approximation errors, but candidate set in Freeway captures 26% of target probability mass. error bound in Theorem 4.2 may become loose as action space grows. Therefore, not sure to what extent these theoretical results can explain empirical performance, especially in large action spaces."

Our theory establishes how target-mass coverage controls CTMC generator approximation error under stated assumptions. Our diagnostics show that candidates can retain high-advantage actions and support effective control despite low coverage. So Theorem 4.2 characterizes generator approximation error, not RL return or a dimension-independent guarantee and its dependence on intermediate state probabilities can make the bound conservative in large action spaces.

We evaluated candidate budgets on Freeway (K=4K=4, five seeds). Normalized generator error measures the relative difference between candidate-restricted and full-action ideal CTMC transition rates. Advantage gap measures the loss in maximum estimated advantage due to candidate restriction, using the same advantage normalization.

BTarget massNormalized generator error ↓Advantage gap ↓Return ↑
320.150.960.7836.9
640.260.730.2938.1
1280.330.640.1738.6
2560.380.550.1139.6
Full1.000041.0

Increasing the budget reduces measured generator error and the loss of estimated advantage while improving returns. At the default B=64B=64, DRIFT achieves 38.1 despite large approximation error. On Space Invaders (K=4K=4), increasing BB from 32 to 256 similarly reduces normalized generator error from 0.91 to 0.62 and advantage gap from 0.43 to 0.07, while improving returns from 33.5 to 43.6.

Exhaustive enumeration is not always preferable. On Space Invaders, B=64B=64 achieves 36.0 and B=256B=256 achieves 43.6 versus 24.6 using all 1,296 actions (Table 8). Changing the candidate set also changes advantage normalization and smoothed reference weights, hence the effective learning target, not just approximation error. The lower full-enumeration return may reflect optimization or value estimation effects, but their causes remain unisolated (Appendix D.5).

To further examine scalability, we extended our candidate-budget experiments to K=8K=8 (five seeds):

Budget BBFreeway ↑Space Invaders ↑
3224.1±0.616.6±1.2
64 (default)26.0±0.818.1±0.9
12827.6±1.420.5±1.0
25628.3±1.122.4±1.7

Increasing BB from 32 to 256 improves returns by 4.2 points on Freeway and 5.8 on Space Invaders, supporting practical scalability without implying Theorem 4.2's bound remains tight as ∣A∣|\mathcal A| grows.

We clarified the scope and limitations of Theorem 4.2 in Section 4.6, added the generator error and advantage gap diagnostics to Appendix D.4 (Candidate coverage), and reported K=8K=8 sensitivity and enumeration analysis in Appendix D.5.

W1. "some concerns about novelty of paper. From my perspective, method seems more like framework that combines several existing techniques. Although authors provide some ablation studies, I do not think they sufficiently demonstrate unique advantages of each component over existing alternatives, or whether combining these components brings additional benefits."

DRIFT is not just a combination of existing techniques. Our key contribution is a CTMC-specific offline-to-online learning framework jointly addressing value-guided policy improvement, preservation of pretrained sampling dynamics, and scalable learning. The integration is not straightforward because unlike conventional policies, CTMCs require valid transition rates and DRIFT learns these from advantage-weighted targets while handling the mismatch between the full action space and the restricted candidate set. Path-space KL preserves pretrained sampling dynamics, while reference-guided candidates focus learning without exhaustive action evaluation. Our new joint ablation demonstrates how combining these components brings additional benefits..

ConfigurationFreeway ↑Space Invaders ↑
Full DRIFT38.1±0.536.0±1.7
Without path-space KL25.9±0.920.8±3.9
Without reference-guided candidates11.1±1.68.0±0.6
Without both9.5±1.46.5±1.1

The combined method achieves the highest returns on both games at K=4K=4, five seeds. More importantly, the components show positive interaction effects of 10.6 points on Freeway and 13.7 on Space Invaders, indicating that path-space KL is more beneficial when combined with reference-guided candidates.

We added the joint ablation to Section 5.1 (Table 3) and interaction analysis to Appendix D.5.

We hope this addresses the reviewer's concerns and are happy to answer further questions.

Dear Reviewer,

As the discussion period is nearing its end, we hope our responses and additional experimental results have addressed your concerns. We would be happy to provide any further clarification if needed and would appreciate your feedback.

Reviewer_mf67OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DRIFT for offline-to-online fine-tuning of discrete CTMC policies. The method combines advantage-weighted discrete flow matching with path-space KL regularization, constraining the entire CTMC path rather than only the action distribution. To scale to large discrete action spaces, the authors construct candidate action sets from reference-policy rollouts and uniform samples, and provide theoretical bounds on coverage error and generator stability. Experiments cover macro-action MinAtar (K=4, K=8), Jericho without the admissible-action oracle, standard MinAtar, and discretized D4RL.

Critical strengths
  • The candidate-set approximation is theoretically supported: Proposition 4.1 shows coverage error decreases exponentially with candidate count, and Theorem 4.2 links target-generator error to the missing target mass.
  • Using path-space KL for CTMC policy fine-tuning is better aligned with the path behavior of discrete generative policies than policy-level KL alone, and the experiments cover combinatorial and language-based action spaces.
Critical weaknesses
  • The path-space KL is implemented with a Monte Carlo plug-in surrogate, where gradients flow only through the evaluated uθ and λθ. The paper acknowledges that this is not an unbiased gradient estimator, which weakens the strict “trust region” interpretation.
  • Empirical conclusions are mixed: under Jericho without the oracle, the absolute score is only 7.3%, and some games are worse than DRRN-Struct/CSM. D4RL discretization is coarse and the candidate-set advantage disappears when |A|=22. Sensitivity to the regularization weight α and candidate budget needs more systematic analysis.
Questions for authors
  1. How does the bias of the path-space KL surrogate affect the final policy? Can the authors provide an unbiased or bias-corrected estimator?
  2. How does the candidate budget relate to target-mass coverage across tasks? Is there a strategy for automatically choosing the candidate count?
  3. Since the candidate-set advantage disappears in the D4RL discretized setting, what do the authors consider the main application boundary of DRIFT?
5 comments
AuthorsOpenReview ↗

Response to Reviewer mf67 (1/2): Q1, W1 (Path-Space KL estimation and regularization)

We sincerely thank the reviewer for their time and helpful feedback, and for recognizing our theoretical contributions, suitability of path-space KL, and diverse experiments. We address your concerns below.

Q1. "How does the bias of the path-space KL surrogate affect the final policy? Can the authors provide an unbiased or bias-corrected estimator?"

The gradient bias can affect the final policy by omitting derivatives through the CTMC state distribution. However, our compensator provides a useful regularization signal, consistent with related rate-based and Bregman KL approaches discussed below. Our new experiments confirm this with 0.9971 median gradient alignment and true KL reduction in all 20 tested configurations reported below. We also implemented an unbiased score-function estimator, which showed substantially higher gradient variance and less consistent KL reduction even with a leave-one-out baseline.

Our compensator estimator provides a useful regularization signal as its local gradient log⁡(uθ/uref)\log(u_\theta/u_{\mathrm{ref}}) pushes transition rates toward the reference even with detached trajectories. Its nonnegative Bregman form is closely related to the KL value estimator proposed by Schulman [1] and used in GRPO [2], which is nonnegative and designed to have low variance. Near the reference, the omitted occupancy-gradient contribution is second-order in the rate deviation under smoothness assumptions, helping explain the observed gradient alignment. Related work uses generalized KL rate matching for CTMC fine-tuning [3], and the same rate-based path-space KL identity [4]. These connections provide theoretical intuition and practical precedent, while our experiments directly validate the gradient behavior of our estimator.

We report evaluations on 20 four-state CTMC configurations against exact path-space KL gradients:

EstimatorCosine ↑Rel. error ↓Norm. ΔKL ↓Variance ↓P[descent] ↑
Exact1.00000%−0.98601.000
Sampled stop-gradN/A100%+0.05531.7910.501
Compensator (ours)0.99718.3%−1.0000.1301.000
Score-function0.99856.9%−0.81935.5530.675
Score-function + baseline0.99856.9%−0.82034.5500.672

Exact-gradient row serves as an oracle reference. Values are medians across configurations. Cosine and relative error compare population-mean gradients with exact gradients. Variance is normalized by the squared exact-gradient norm, and P[descent] measures the fraction of sampled gradients aligned with the exact gradient.

The compensator reduced true path-space KL in all 20 configurations, versus 4/20 for the sampled estimator. Its relative gradient error ranged from 6.0% at deviation 0.5 to 15.7% at deviation 3.0, while maintaining successful KL reduction across all tested deviation levels.

For the score-function-corrected estimator (1+L^)∇θL^(1+\hat L)\nabla_\theta\hat L, which is unbiased under exact CTMC sampling, together with a leave-one-out control variate, its gradient variance was 273.6 times that of the compensator, or 265.9 times with the control variate. These results support the compensator's practical bias–variance trade-off.

We clarified the practical KL estimator in Section 4.4 and added its derivation, related literature, gradient analysis, and numerical comparison in Appendix C.2 of the revised manuscript.

W1. "The path-space KL is implemented with a Monte Carlo plug-in surrogate, where gradients flow only through the evaluated uθ and λθ. The paper acknowledges that this is not an unbiased gradient estimator, which weakens the strict “trust region” interpretation."

DRIFT regularizes the exact CTMC path-space KL through its compensator representation. The approximation lies in the gradient, since sampled CTMC states are detached. This retains a meaningful rate-based KL regularization signal without differentiating through the full state occupancy distribution which is computationally expensive in large discrete action spaces, while unbiased Monte Carlo estimators can suffer from high variance as we have seen in Q1.

Our existing RL ablations further show improved returns with regularization on Freeway (38.1 vs. 25.9) and Space Invaders (36.0 vs. 20.8). These results support its effectiveness as a practical KL regularizer, although the approximate gradient does not establish a strict trust-region guarantee. We have clarified in Section 4.4 of the revised manuscript that our KL penalty provides practical regularization without guaranteeing a strict bound on path divergence.

References

[1] Schulman. "Approximating KL Divergence." Technical blog, 2020.

[2] Shao et al. "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models." arXiv:2402.03300, 2024.

[3] So et al. "Discrete Adjoint Matching." ICLR 2026.

[4] Tang et al. "TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusion." arXiv:2509.25171, 2025.

AuthorsOpenReview ↗

Response to Reviewer mf67 (2/2): Q2, Q3, W2 (Candidate coverage, application scope, and sensitivity)

Q2. "How does the candidate budget relate to target-mass coverage across tasks? Is there a strategy for automatically choosing the candidate count?"

The required candidate budget depends primarily on target-policy concentration rather than action-space size alone. An adaptive strategy could increase the budget until the marginal improvement in estimated target-mass coverage no longer justifies the additional computation.

Specifically, Proposition 4.1 establishes that the target policy approximation error equals 2ϵ(s)2\epsilon(s), where ϵ(s)\epsilon(s) is the excluded target mass. Under the idealized independent sampling model, its expectation decreases exponentially with the number of sampled candidates.

Empirically, increasing BB from 32 to 256 improves Freeway target-mass coverage from 15% to 38% and returns from 36.9 to 39.6 (Appendix D.4). Jericho similarly shows improved candidate recall and returns with larger budgets (Table 17).

While these results support the computation-coverage trade-off, estimating target-mass coverage without exhaustive action evaluation remains challenging. We therefore present adaptive budget selection as a promising heuristic.

We have clarified the relation between candidate budget and coverage in Appendix D under Candidate coverage and corrected the uniform sampling term in Proposition 4.1 and its proof.

Q3. "Since candidate-set advantage disappears in D4RL discretized setting, what do authors consider the main application boundary of DRIFT?"

W2a:" D4RL discretization coarse, candidate-set advantage disappears when |A|=22."

In the D4RL setting with only 22 discretized actions, candidate-set approximation offers no computational advantage because all actions can be evaluated efficiently. Still, DRIFT achieves the highest average final normalized score across six tasks (24.4 vs. 20.6 for the next best method, PEX, Table 21).

These results show that DRIFT remains effective even in small discrete action spaces, while its candidate-set mechanism is most beneficial in large or combinatorial action spaces where exhaustive evaluation becomes computationally expensive. We have clarified this application boundary in Section 5 under Additional results and takeaways and in Appendix F.2 under Limitations of the Discretized Setting.

W2. "Empirical conclusions mixed: Jericho without oracle absolute score 7.3%, some games worse than DRRN-Struct/CSM. Sensitivity to regularization weight α and candidate budget needs more systematic analysis."

Jericho's oracle-free setting is challenging, where DRIFT achieves highest mean normalized score among evaluated oracle-free methods (7.3% vs. 3.6% for the strongest baseline), although DRRN-Struct performs slightly better on Detective and CSM on Ludicorp. These results highlight the difficulty of oracle-free action generation and the remaining room for improvement.

Regarding sensitivity, Appendix D.4-D.5 reports sweeps over α\alpha and candidate budgets at K=4K=4. On Freeway, α=0.01\alpha=0.01 achieves 38.1 versus 25.9 without regularization, while increasing BB from 32 to 256 improves returns from 36.9 to 39.6. We further conducted sensitivity analyses on Freeway and Space Invaders at K=8K=8, five seeds.

Regularization weight α\alpha (K=8K=8):

α\alphaFreeway ↑Space Invaders ↑
017.2±1.110.6±0.9
0.00121.6±1.014.0±1.5
0.01 (default)26.0±0.818.1±0.9
0.0323.8±0.816.7±1.3
0.120.7±1.313.1±0.9

Candidate budget BB (K=8K=8):

BudgetFreeway ↑Space Invaders ↑
3224.1±0.616.6±1.2
64 (default)26.0±0.818.1±0.9
12827.6±1.420.5±1.0
25628.3±1.122.4±1.7

The new results confirm similar trends at K=8K=8: α=0.01\alpha=0.01 achieves the highest mean returns among the tested weights, while removing regularization substantially reduces performance. Increasing BB from 32 to 256 improves returns from 24.1 to 28.3 on Freeway and from 16.6 to 22.4 on Space Invaders. These results demonstrate the importance of path-space regularization and the benefit of larger candidate budgets, while highlighting sensitivity to α\alpha and the trade-off between candidate budget and computation. We have added the K=8K=8 sensitivity results to Appendix D.5 under Ablation Studies and summarized them in Section 5 under Ablations.

We hope this addresses the reviewer's concerns and are happy to answer further questions.

Dear Reviewer,

As the discussion period is nearing its end, we hope our responses and additional experimental results have addressed your concerns. We would be happy to provide any further clarification if needed and would appreciate your feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.