acceptodds
Under review as a conference paper at ICLR 2027

Occupancy Modeling via Drifting: One-Step Generation Enables Direct Policy Extraction

Abstract

The growing interest in behavior foundation models, combined with recent advancements in generative modeling, has led to an appealing recipe for reinforcement learning from unlabeled data: pre-train a generative model to model occupancies, i.e., discounted state visitations, on reward-free trajectories, such that a policy can be easily extracted once a small reward-labeled dataset becomes available. We decompose this recipe along three axes: (i) the degree to which the multi-modal structure of occupancies must be captured, (ii) the class of generative model, and (iii) the method used for policy extraction. Along each axis, we consider existing choices, propose simpler alternatives, and validate each decision experimentally. This analysis culminates in a novel, streamlined method that leverages drifting, a modern one-step generative sampler, to model the occupancy. We show that this enables policy extraction via direct differentiation through the occupancy model, and bypasses the need for critic distillation. Overall, we observe that this nimble setup leads to improvements in performance on standard benchmarks.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, drifting, deep reinforcement learning, unsupervised reinforcement learning, offline reinforcement learning, generative model, occupancy model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LyUyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TD-Drift, a temporal-difference objective for learning occupancy distributions with one-step drifting models from reward-free data. The method enables efficient occupancy sampling and direct policy extraction without critic distillation. Experiments demonstrate improvements in performance and policy extraction efficiency over existing flow-based occupancy methods.

Critical strengths

1.The paper introduces a temporal-difference objective that extends drifting to occupancy modeling, providing an efficient one-step alternative to flow-based methods.

Critical weaknesses
  1. Direct policy extraction appears to be a natural consequence of using a differentiable one-step generative model. The paper also lacks comparisons with other one-step generative approaches, such as MeanFlow [1][2], making it unclear what advantages are specific to TD-Drift.
  2. The paper emphasizes efficient downstream adaptation through reward-free pre-training, yet policy extraction still requires 500K reward-labeled transitions. Without an analysis of reward-label efficiency or comparisons with standard offline RL methods trained on the same labeled data [3].

[1]Nguyen X T, Yoo C. One-step flow q-learning: Addressing the diffusion policy bottleneck in offline reinforcement learning[C]//International Conference on Learning Representations. 2026.

[2]Geng Z, Deng M, Bai X, et al. Mean flows for one-step generative modeling[J]. Advances in Neural Information Processing Systems, 2026.

[3]Park S, Li Q, Levine S. Flow q-learning[J]. arXiv preprint arXiv:2502.02538, 2025.

Questions for authors
  1. Figure 7 reports a 2.4x speedup in policy extraction for (In)DOM-Direct over (In)FOM, while Table 3 shows that (In)DOM requires more pre-training steps. How does the total computational cost compare when both pre-training and policy extraction are considered?

I would consider raising my score if the authors could provide convincing evidence of TD-Drift's advantages over other one-step generative methods and demonstrate its reward-label efficiency under fair comparisons with offline RL baselines.

1 comment

We thank the reviewer for their constructive feedback and clear re-evaluation criteria.

Comparison to other one-step generative approaches

Applying other one-step generative models such as MeanFlow to occupancy modeling is not straightforward, as standard training objectives require data samples alone, and temporal-difference variants have not been formulated yet to the best of our knowledge. MeanFlow regresses interval-averaged velocities, and we believe that deriving a Bellman-consistent bootstrap target for it would be a contribution in its own right. We thus focus on showing that a class of one-step generative models is competitive with the multi-step models so far used for occupancy modeling, and we do not claim that drifting represents the optimal model class. Having established that one-step models are competitive, we believe that an exploration of other frameworks is very interesting. We, however, consider this beyond the scope of this paper and have added it as direction for future work.

Reward-label efficiency and offline RL baselines

We thank the reviewer for these suggestions. We trained an offline RL method (TD3+BC[1] ) on reward-labeled data alone and compared it with DOM-Direct and InDOM-Direct at different data budgets [10K, 100K, 200K , 500K], and we have added this comparison to the paper (Appendix E.3). For TD3-BC we tuned the BC coefficient and found 2.5 being the best (as also reported in [2]). We find that while offline RL methods increase performance with the data budget, occupancy-based methods outperform them in all data regimes. This further motivates reward-free pretraining, and strengthens the relevance of our method.

Computational cost

We focused on timing policy extraction, as the pre-trained occupancy model can be reused for multiple rewards, and its cost amortized. As you suggested, we can nevertheless report the computational cost of pre-training for drifting models (6.37ms per training step), and compare it to that of flow-based models (2.23ms per training step). All experiments run on a single Nvidia A100. Drifting is thus generally more compute intensive, and requires more gradient steps to converge. We believe that this is a fair cost to pay in exchange for generally faster policy extraction, which is only paid once since the pre-trained occupancy model can be reused for all rewards. We have added this discussion and table with total cost as Appendix E.4. On a side note, we found and corrected an error in our timing analysis for policy extractions, and updated it accordingly. While all drifting-based approaches offer significantly faster (>2x) policy extraction than their flow-based counterparts, InDOM-Direct requires larger forward passes and is 1.2x faster.


Thank you for the clear and actionable feedback. We hope we were able to address each comment in full, and strengthen our submission accordingly. We remain available for further discussion.

[1] Fujimoto et al. “A minimalist approach to offline reinforcement learning.” NeurIPS 2021

[2] Yarats et al. "Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning."

Reviewer_iNKROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies pre-training an occupancy model (the discounted distribution over future states) on reward-free data, then extracting a policy once a small reward-labelled set is available. It decomposes the pipeline along three binary choices: conditional (latent intention) versus marginal occupancy; flow matching versus a one-step drifting generator; and policy extraction through a distilled critic versus directly. It introduces TD-Drift, a drifting objective whose target mixes the next state with a bootstrap from the model's own target network (Eqs. 10-12), and a direct extraction rule that differentiates the Monte Carlo value through the one-step generator (Eq. 15). Six variants are evaluated on four DeepMind Control domains (16 tasks) with ExORL data, against behavior cloning, FB and InFOM. The paper reports that drifting is competitive with flow at lower cost, that intention conditioning helps but is not necessary, and that direct extraction matches or improves critic-based extraction and is about 2.4 times faster.

Critical strengths
  1. A clear decomposition of the pipeline. The three choices are isolated and each is tested, which turns entangled design decisions into a map. This is consequential because practitioners can see which choices matter, and the paper is easy to follow.
  2. Direct policy extraction is simple. Differentiating the value through a one-step generator removes critic distillation and, for the marginal variant, the intention encoder, the KL weight and the search over latents (Section 5.5). This is consequential because it shortens the pipeline, and the paper reports about 2.4 times faster extraction (Figure 7, extrapolated).
Critical weaknesses
  1. The contribution is a recombination with little new knowledge. By the paper's own account, TD-Drift builds on TD-Flow's and InFOM's construction with the drifting loss in place of the flow loss (Section 4.2, Appendix A), and direct extraction is the chain rule through a one-step generator (Eq. 15). The main findings come from toggling three existing binary choices and reporting the best combination. No analysis supports the "Bellman-consistent" claim: the text states no convergence result for the bootstrapped target, and there is no ablation without bootstrapping. This is consequential because a reader learns which combination worked on four domains, but not why, and the generality of the recipe rests on empirical evidence alone.

  2. The evidence does not support the claims, and the setting does not test the motivation.

    • Scope: four DMC domains (16 tasks) with one exploration dataset. InFOM, the base pipeline, reports 36 state-based and 4 image-based tasks (per its public abstract).
    • Selection: each method is reported at its best configuration per domain, with configurations scored on the same four tasks that are reported (Table 1 caption; Appendix C.1). Pre-training budgets differ (1.5M steps for cheetah and 2M elsewhere for drifting, 1M for flow; Table 3), so the gain cannot be attributed to drifting.
    • Headline: the average DOM-Direct 375 versus InFOM 326 is driven by quadruped (682 versus 458). Without quadruped it is 272 versus 281 (computed from Table 1), and DOM-Direct is behind InFOM on cheetah and jaco. FB averages 503, above every occupancy variant, yet the abstract claims improvements only relative to InFOM. The averages carry no intervals.
    • Motivation: the evaluation tasks have environment rewards available. No experiment shows a reward-missing scenario, a few-label regime, or reuse of one model across many reward functions, and there is no offline RL baseline trained directly on the same 500K labelled transitions.

    This is consequential because the improvement may reflect budget and per-domain selection, and the evidence does not show when occupancy pre-training is needed at all.

Questions for authors
  1. Can you provide flow and drift results with equal pre-training budgets and equal tuning effort, so the gain can be attributed to drifting rather than to the longer pre-training in Table 3?
  2. Are configurations selected on the same tasks that are reported (Appendix C.1)? Please give results with configurations selected on held-out tasks or seeds.
  3. Please give per-domain results with confidence intervals, with FB included in the comparison. How should the improvement over InFOM be stated, given that quadruped drives the average and FB is higher?
  4. What keeps the bootstrapped drifting target stable? Can you show an ablation with gamma at zero, and any analysis or diagnostics of the fixed point? What does flow with direct extraction give, for example through a differentiable solver?
  5. What is the total cost, pre-training plus extraction, for each method, on the same hardware? Please report GPU type and count, batch size, the number of samples per value estimate, wall-clock for pre-training, and whether code will be released.
  6. Why do BC and FB score near zero on jaco (3 and 2, against 161 to 201)? Is FB tuned there, given that the codebase was extended by the authors (Appendix C.1)?
3 comments

We thank the reviewer for their constructive feedback and we address main (W)eaknesses and (Q)uestions below.

W1: Contribution

While TD-Drift relates to TD-Flow and builds upon bootstrapping formulations, drifting models involve a fundamentally different objective, and hence the flow loss cannot be trivially replaced with the drifting loss. TD-Drift thus needs to design an attraction field as in Eq. 11. We acknowledge that direct extraction is an application of the chain rule, and we believe that its value lies precisely in its simplicity compared to existing pipelines involving distillation. To the best of our knowledge, the three choices we validate empirically have either been only marginally explored in generative occupancy modeled or are not existing: previous works such as TD-Flow and InFOM only consider flow matching, do not evaluate direct policy extraction, and only briefly consider marginal modeling (InFOM, Figure 5). We believe there is value in exploring simpler alternatives and clarifying when they work. We appreciate the suggestion to introduce an analysis of Bellman consistency and convergence. We have done so in new Appendix F and refer to the response to Q4.

W2.1: Scope:

Our contribution is not only methodological, but also largely focuses on systematic empirical validation of design choices. Each variant and baseline we considered was thus extensively tuned. Considering computational constraints, we thus chose to prioritize a rigorous fine-grained evaluation over a broad sweep over environments.

W2.2/Q1: Pre-train steps mismatch

We thank the reviewer for pointing this out. We re-trained InFOM with the same number of pre-training steps as InDOM (Table 3), while also matching the normalization scheme and freezing the occupancy after pre-training, as suggested by reviewer pbpw. This fully aligns drifting- and flow-based pipelines. We report new and old values in the table below. Its performance does not change significantly, so the gains we report come from the method rather than differences in the pipeline, such as longer pre-training. We have updated the results in the paper accordingly.

DomainInFOM (matched steps, normalization, frozen after pre-training)InFOM (old protocol)
cheetah284.5 ± 25.9304.6 ± 13.3
quadruped425.6 ± 64.2458.4 ± 36.9
walker390.0 ± 8.8364.4 ± 7.2
jaco179.3 ± 1.5175.0 ± 2.5

W2.3/Q3: Please give per-domain results with confidence intervals, with FB included in the comparison

Thank you for this suggestion. Table 1 now includes both per-domain and aggregate results with confidence intervals for all methods.

W2.3/Q3: How should the improvement over InFOM be stated, given that quadruped drives the average and FB is higher?

We believe that the nuanced comparison between DOM-Direct and InFOM shows the importance of the breakdown of each method in Section 5.1, 5.2, and 5.3, and is acknowledged explicitly on line 453. DOM-Direct’s appeal is its simplicity: it removes the critic, the intention encoder, the KL weight, and the latent search, while remaining competitive on average, largely thanks to quadruped. We note that Table 1 however, recommends another recipe, namely InDOM-Direct, which outperforms InFOM in each environment. Our main conclusion is thus that, while either framework may be preferable in a given environment, drift-based alternatives have a higher aggregated return than flow-based methods. FB is not a generative occupancy model and follows a different, zero-shot protocol. We include it as a reference point for occupancy-based methods in general, and its results (strong on cheetah and walker, weaker on quadruped and jaco) show that neither family dominates. Our claims concern the generative occupancy pipeline. We have revised the abstract to state the improvement relative to generative occupancy models explicitly.

W2.4: No experiment [...] shows reuse of one model across many reward functions

All occupancy-based methods in our paper reuse a single pre-trained occupancy model across all reward functions of a domain. Only the policy is fitted for each reward function revealed at test time (FB, being zero-shot, doesn’t require fitting a policy). Rewards are never used during pre-training. We have added this clarification on line 336.

W2.4: Reward-label efficiency and offline RL baselines:

We thank the reviewer for these suggestions. We trained an offline RL method (TD3+BC [1]) on reward-labeled data alone and compared it with DOM-Direct and InDOM-Direct, at different data budgets [10K, 50K, 500K]. For TD3-BC we tuned the BC coefficient and found 2.5 to be the best (as also reported in [2]). We find that while offline RL methods increase performance with the data budget, occupancy-based methods outperform them in all data regimes. This further motivates reward-free pretraining and strengthens the relevance of our method. We have added this comparison to the paper (Appendix E.3).

Q2: Are configurations selected on the same tasks that are reported (Appendix C.1)? Please give results with configurations selected on held-out tasks or seeds.

We thank the reviewer for carefully analysing our evaluation protocol. Configurations were selected per domain on the mean return over a single task (as reported in line 840) using three tuning seeds. Reported results use five seeds, which include these three. To remove the optimistic bias, we will re-evaluated each method using 5 held-out seeds. Due to computational costs, we will update the manuscript with these results for the camera-ready revision.

W1/Q4: What keeps the bootstrapped drifting target stable? Can you show an ablation with gamma at zero, and any analysis or diagnostics of the fixed point?

Thank you for this suggestion. We added a fixed-point analysis in Appendix F.1. There we show that the true occupancy mπm^\pi is a stationary point of TD-Drift: if the target network matches mπm^\pi, the mixture in Eq. 10 equals mπm^\pi by the Bellman equation (Eq. 2). The drifting field (Eq.4) then vanishes because the generated distribution matches its target, and the gradient vanishes as well (Eq. 24). This establishes that mπm^\pi is consistent with the TD-Drift objective. In practice, we stabilize the bootstrap target with several mechanisms: a warmup phase and median temperature rescaling (Appendix E) and decoupled noise (Appendix D), a slowly updated target network and a conservative learning rate (Table 3). An ablation with γ=0\gamma=0 removes bootstrapping and reduces the model to a next-state predictor. The occupancy model could not support long-horizon value estimation, and thus it would render the method ineffective.

Q4: What does flow with direct extraction give, for example through a differentiable solver?

Following the reviewer’s suggestion, we also ran a FOM-Direct baseline that extracts the policy by backpropagating through the ODE solver:

DomainFOM-DirectFOMDOM-Direct
cheetah344.7 ± 12.0286.0 ± 12.4256.4 ± 46.0
quadruped539.0 ± 13.1538.2 ± 8.4662.3 ± 10.8
walker551.3 ± 7.7500.5 ± 8.2413.6 ± 12.9
jaco207.6 ± 2.9153.0 ± 1.5154.8 ± 3.4

Interestingly, FOM-Direct outperforms FOM, despite prior reports that gradients through iterative samplers are unstable and led to poor gradient estimates [3, 4, 5]. However, it is slower per update than DOM-Direct, since each gradient step backpropagates through all integration steps. Finally, it remains below InDOM-Direct. Understanding why gradients through the flow sampler are informative in this setting, despite prior reports, is an interesting direction for future work.

Q5: What is the total cost, pre-training plus extraction, for each method, on the same hardware?

All experiments run with hyperparameters described in Table 3. We focused on timing policy extraction, as the pre-trained occupancy model can be reused for multiple rewards, and its cost amortized. Nevertheless, as suggested, we have added the computational cost of pre-training for drifting models (6.37ms per training step), and compared to that of flow-based models (2.23ms per training step) to Appendix E.4. All experiments run on a single Nvidia A100. Drifting is thus generally more compute-intensive and requires more gradient steps to converge. We believe that this is a fair cost to pay in exchange for generally faster policy extraction, which is only paid once since the pre-trained occupancy model can be reused for all rewards. On a side note, we found and corrected an error in our timing analysis for policy extractions, and updated it accordingly. While all drifting-based approaches offer significantly faster (>2x) policy extraction than their flow-based counterparts, InDOM-Direct requires larger forward passes and is 1.2x faster. Our code is already available on the anonymous project page (https://sites.google.com/view/td-drift/).

Q6: Why do BC and FB score near zero on jaco (3 and 2, against 161 to 201)? Is FB tuned there, given that the codebase was extended by the authors (Appendix C.1)?

BC performance is highly dependent on whether the dataset contains trajectories that solve the task. All datasets are collected via highly explorative policies, and in Jaco they might not contain trajectories solving the reaching tasks, so imitation leads to near-zero return. FB can only optimize rewards that are linear in its learned features and its performance depends heavily on the feature and subsequently data quality. FB is known to struggle with manipulation tasks such as Jaco [1]. The results were obtained by tuning over discount factor and orthonormalization coefficient as reported in Appendix C.1. The results match those reported in previous work [2]


We thank the reviewer for the detailed and actionable feedback, which we have integrated in the manuscript. In particular, we believe that the new analysis of Bellman consistency, and the comparison with offline RL have strengthened our submission. Please, let us know if these new results can support an increase in score, and if you have any remaining questions.

[1] Mohan et al. "Is Manipulation Hard for Successor Measures?" 3rd RLBrew Workshop at RLC 2026

[2] Jeen et al. "Zero-shot reinforcement learning from low quality data." NeurIPS 2024

[3] Zheng et al. “Intention-conditioned flow occupancy models.” ICLR 2026

[4] Park et al. “Flow q-learning.” ICML 2025

[5] Zhang et al. “SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modeling.” ICLR 2026

Reviewer_pbpwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper follows the two-stage pipeline of InFOM (Zheng et al., 2026) for reinforcement learning from reward-free data. First, a generative model of the occupancy (the discounted distribution of future states) is pre-trained on reward-free transitions and then frozen. Second, once a reward-labeled dataset is available, a policy is extracted from it. The paper breaks this pipeline down along three axes:

  • conditioning: whether the occupancy is conditioned on latent intentions inferred by a variational encoder, or modeled marginally;

  • model class: TD-Flow-style flow matching or drifting models (Deng et al., 2026);

  • policy extraction: either distill a critic from Monte Carlo value estimates, or extract directly by backpropagating the Monte Carlo value estimate through the one-step generator into a DDPG+BC actor update (Eq. 15).

The main technical contribution is TD-Drift. It inserts the Bellman mixture of the observed next state and bootstrap samples from a target network into the attraction term of the drifting field (Eqs. 10–12 and 16). Practical components support it (Appendices D–E): decoupled noise, a warm-up phase, tanh⁡\tanh outputs with min-max normalization, and median-based temperature rescaling. All six combinations are evaluated on 16 tasks from four DeepMind Control domains (ExORL RND data, five seeds), with behavior cloning and forward-backward representations as references. The paper reports that drifting is competitive with flow matching, that intention conditioning helps modestly, that direct extraction matches or beats critic-based extraction in seven of eight settings, and that policy extraction with drift models is 2.1–2.4$\times$ faster in wall-clock time.

Critical strengths
  1. A simple, practical route to critic-free policy extraction from a generative occupancy model.

    • What it is: TD-Drift gives a one-step reparameterized sampler of future states, so the Monte Carlo value estimate can be differentiated with respect to the action (Eq. 15). This removes the critic-distillation stage that the flow-based InFOM pipeline needs (p. 6, ll. 284–290).

    • Evidence: Both extraction routes use the same drift pre-training pipeline and the same γ\gamma/κ\kappa grid (App. C, Table 4). Direct extraction matches or improves on critic-based extraction in seven of eight domain/variant settings (Fig. 6). From the Table 2 means I compute average gains of +7% for InDOM-Direct over InDOM and +5% for DOM-Direct over DOM. Direct extraction also cuts extraction wall-clock time by about 2.4×2.4\times relative to (In)FOM (Fig. 7). The best variant, InDOM-Direct, beats both flow-based variants in every domain (Table 2); its four-domain mean is 400, against 326 for InFOM (Table 1) and 339.5 for FOM (my computation from Table 2).

    • Why it matters: it shows that an expressive generative occupancy model and a cheap first-order extraction step can be combined.

  2. A transparent, internally consistent design-space study with reusable practical findings.

    • Consistency: all six combinations are reported per task (Table 2). I recomputed every domain mean and every percentage in Figs. 3, 5 and 6 from Table 2, and they agree with Table 1 up to rounding.

    • Honest reporting: the paper shows where its method loses: jaco in Fig. 3, cheetah for DOM-Direct in Fig. 6, and the stronger FB reference on cheetah and walker in Table 1.

    • Reusable findings: it documents three failure modes and their fixes: instability from noise coupling (Fig. 8, p. 18), collapse to BC-level returns without a warm-up phase (Fig. 9), and per-domain distance scales handled by median temperature rescaling (Fig. 10). Limitations are stated plainly (p. 9, ll. 470–479).

    • Why it matters: even if the headline gains turn out to be modest, these findings are useful to anyone training one-step generative models with TD targets.

Critical weaknesses
  1. The comparisons behind the main flow-vs.-drift attributions are not controlled.

    • The claim: the paper says InFOM and InDOM “differ only in the generative model” (p. 7, ll. 349–350), and that all variants “differ only in the studied components” (p. 6, last line, just below l. 323).

    • Training setup differs. The drift models use 1.5M–2M pre-training steps instead of 1M, a learning rate of 10−410^{-4} instead of 3×10−43\times10^{-4}, and a target update rate of 10−410^{-4} instead of 5×10−45\times10^{-4} (Table 3). They use min-max normalization with tanh⁡\tanh outputs, while the flow models use mean/std normalization with a box projection (p. 17, l. 908 to p. 18, l. 921). They add a 10K-step warm-up phase (p. 20, ll. 1041–1043).

    • Hyperparameters differ. Table 4 gives λKL∈{10,100}\lambda_{\mathrm{KL}}\in\lbrace 10,100\rbrace for the drift models against {0.005,…,0.2}\lbrace 0.005,\dots,0.2\rbrace for InFOM, dim⁡(z)=128\dim(z)=128 against 512 on walker and jaco, and α=0.3\alpha=0.3 against 0.10.1 on jaco.

    • Tuning budgets differ. The drift variants were searched over γ\gamma (2 values) and κ\kappa (3 options), plus λKL\lambda_{\mathrm{KL}} (4 values) for the intention variants. The flow baselines had only γ\gamma re-tuned (p. 16, ll. 830–832 and 842–844). Selection used evaluation returns averaged over three seeds (p. 16, ll. 847–850).

    • Inherited InFOM settings come from a different protocol. The authors argue that InFOM’s settings were already tuned by its authors (p. 16, ll. 827–829), which is reasonable. But in InFOM, the flow occupancy model and the intention encoder keep training during the 500K extraction steps (stated here on p. 16, ll. 835–838; see also Algorithm 2 and App. D.4 of Zheng et al.). Here, the drift models are frozen. The paper says in general terms that the occupancy model is frozen (p. 4, Sec. 4; p. 7, ll. 334–335), but also that InFOM’s pipeline was rerun with its reported configurations (p. 16, ll. 857–858). So it is unclear whether the InFOM/FOM reruns also update the flow model during extraction.

    • Why it matters: finding (1) (p. 2, ll. 090–097), the +14% in Fig. 3 and the comparison with InFOM in Table 1 all attribute differences to the model class. The drift side gets longer pre-training, different normalization and a larger selection budget, so these comparisons cannot isolate the effect of drifting. The extraction comparison (Fig. 6) is cleaner because both arms come from the drift pipeline, and I consider it the most reliable result.

    • Fixability: likely fixable. The authors could run flow baselines with the same pre-training steps and normalization (or show learning curves that saturate by 1M steps), state explicitly what is frozen, and report the number of configurations evaluated per method.

  2. The evidence is narrow, and the average advantage of the simplest recipe depends on one domain.

    • Scope: the evaluation covers one benchmark family (four DMC domains, 16 tasks) and one dataset type (1M RND transitions; p. 7, ll. 329–337). The abstract nevertheless claims improvements “on standard benchmarks”. InFOM itself was evaluated on 36 state-based and 4 image-based tasks (Zheng et al., abstract).

    • DOM vs. FOM is the cleanest unconditional model-class comparison. From Table 2 it gives −0.3% on cheetah, −20% on walker, +48% on quadruped and −30% on jaco.

    • DOM-Direct vs. FOM: DOM-Direct is the “no intention encoder, no KL weight” recipe highlighted in Sec. 5.5 (p. 9, ll. 451–453). Against FOM it gives −10%, −16%, +54% and +5% (same domain order). Without quadruped, its three-domain mean (272) is below both InFOM (281) and FOM (305).

    • Table 1 and Sec. 5.5 compare only against InFOM (mean 326). FOM is the stronger flow variant (mean 339.5 from Table 2) and the natural counterpart of DOM-Direct.

    • On quadruped, both flow baselines score below the BC reference: InFOM 458±37458\pm37 and FOM 443±9443\pm9, against BC 501±3501\pm3.

    • In fairness: InFOM’s own published quadruped returns are also below every drift variant (585–682). These come from Table 4 of Zheng et al., under their protocol; the four-task mean is ≈512\approx 512 by my computation. So the quadruped advantage itself looks real.

    • Efficiency is shown for extraction only. The claims “computationally more efficient” (p. 9, ll. 468–469) and “nimble” (abstract) rest on Fig. 7. Fig. 7 measures policy extraction only, is extrapolated from 10K steps, and gives no hardware. Drift pre-training uses 1.5–2$\times$ as many steps (Table 3) and is acknowledged to cost more per step (p. 9, ll. 470–472), but its cost is not reported.

    • The FB reference has the highest mean in Table 1 (503 vs. 400), even though it adapts zero-shot (p. 16, ll. 858–860).

    • Why it matters: the paper’s general recommendation (p. 2, ll. 094–097) is drifting plus direct extraction, potentially without intentions. On this evidence it is robustly supported only for InDOM-Direct, and only on this benchmark.

    • Fixability: partly fixable. The authors could add FOM to Table 1, report pre-training and total compute, and evaluate on further data regimes, e.g., the OGBench tasks used by InFOM.

Minor points.

  • Fig. 4: the text says that the return of InDOM “grows with λKL\lambda_{\mathrm{KL}}” in every domain (p. 7, ll. 365–366), but on cheetah it falls between λKL=10\lambda_{\mathrm{KL}}=10 and 100100 (roughly 330 to 290, read from the plot). For three domains the selected value, λKL=100\lambda_{\mathrm{KL}}=100, is the edge of the grid (Table 4).

  • In Fig. 4, the quadruped point at λKL=100\lambda_{\mathrm{KL}}=100 is roughly 650. Table 2 reports 585±4585\pm4 for InDOM on quadruped, which also uses λKL=100\lambda_{\mathrm{KL}}=100 (Table 4). In addition, App. C says λKL\lambda_{\mathrm{KL}} was swept on a single task per domain (p. 16, ll. 840–842), while C.1 says configurations are scored by four-task means (ll. 847–849).

  • Missing details: the PDF does not give GG or N+N^{+} (Alg. 1), the network architectures, or the hardware. It is also unclear how the drift-specific α\alpha (jaco), M=1M=1 (quadruped) and dim⁡(z)\dim(z) were chosen; none of them is in the stated tuning grid.

  • FB’s jaco score (2±12\pm1, below BC’s 3±13\pm1) comes from the authors’ own extension of the TD-JEPA codebase (p. 16, ll. 862–864). A sanity check would help, since it affects FB’s mean and the claim that FB “falls behind” on jaco (p. 9, ll. 457–458).

Questions for authors
  1. [Clarification] Frozen or fine-tuned baselines, and the size of the gap.

    • During the 500K extraction steps, do the InFOM/FOM reruns keep updating the flow occupancy model and the intention encoder, as in Algorithm 2 of Zheng et al.? Or are they frozen like the drift models?

    • Do the (In)FOM runtimes in Fig. 7 include such updates?

    • Why do the reruns differ from the per-task numbers that Zheng et al. publish (their Table 4), and in both directions? Cheetah walk is 498±14498\pm14 here vs. 254.8±158.6254.8\pm158.6 there. Quadruped stand is 584±112584\pm112 here vs. 718.3±18.7718.3\pm18.7 there.

    • Effect on my assessment: if frozen flow baselines with matched normalization and matched pre-training steps keep a gap similar to Figs. 3 and 5, Weakness 1 would largely go away and I would raise my score. Flow baselines shown to have saturated at 1M steps would count as matched. If the gap shrinks substantially, finding (1) should be weakened.

  2. [Clarification] Tuning budget and seeds.

    • How many configurations were evaluated per method in total? Please include how α\alpha, MM and dim⁡(z)\dim(z) were chosen for the drift variants.

    • Were the five final seeds disjoint from the three tuning seeds?

    • Effect on my assessment: disjoint seeds and comparable budgets would strengthen the results. Overlapping seeds combined with much larger selection budgets for the drift variants would lead me to discount the smaller margins. Examples are InDOM vs. InFOM on cheetah (310±13310\pm13 vs. 305±13305\pm13) and InDOM-Direct vs. FOM on walker (474±13474\pm13 vs. 466±7466\pm7).

  3. [Clarification] What explains the benefit of intentions at λKL=100\lambda_{\mathrm{KL}}=100?

    • This value is the edge of the grid, and the paper itself hypothesizes that such a latent “carries almost no information at all” (p. 7, ll. 371–372). Yet InDOM beats DOM by 19% on walker and 41% on jaco (Fig. 5).

    • Possible confound: the conditional variants use more optimistic aggregation. InDOM’s critic uses an expectile with τ=0.9\tau=0.9, while DOM uses MSE (p. 6, ll. 312–313; Table 3). InDOM-Direct takes a maximum over M=10M=10 sampled latents. Quadruped uses M=1M=1, and it is the only domain where InDOM-Direct trails DOM-Direct (592 vs. 682).

    • Could you report DOM with τ=0.9\tau=0.9, InDOM with zz fixed at extraction, and a measure of how sensitive fθf_\theta’s output is to zz?

    • Effect on my assessment: if the gains survive these controls, finding (2) is supported as stated. If they vanish, the gains should be attributed to optimistic aggregation rather than to intentions.

  4. [Clarification] Consistency of the practical estimator (Eq. 16).

    • The paper says Eq. 16 is a consistent estimator of Eq. 11 under deterministic transitions as N+→∞N^{+}\to\infty (p. 17, ll. 903–905).

    • However, Eq. 16 uses one dataset action a′a' in place of the expectation over a′∼π(⋅∣s′)a'\sim\pi(\cdot\mid s'), and that expectation sits inside a ratio. Averaging the per-a′a' fields over the data is therefore not, in general, the field of the mixture, unless the next action is deterministic. TD-Flow does not have this issue, because its regression loss is linear in the target.

    • Is mπβm^{\pi_\beta} a fixed point of the practical objective under a stochastic behavior policy, such as the one that generated the RND data?

    • Please qualify the claim, or show that the bias is negligible. One option is Q-estimation error against Monte Carlo returns on held-out trajectories; another is a small environment with a known occupancy. Either would also give a direct measure of occupancy quality, which is currently assessed only through downstream return.

    • Effect on my assessment: a convincing answer would raise my soundness rating.

  5. [Suggestion / future work] Isolating the role of one-step generation.

    • The title credits one-step generation for direct extraction. However, reparameterized one-step occupancy samplers already exist: the GAN and normalizing-flow γ\gamma-models of Janner et al. (2020), cited on p. 3 and p. 15. The paper also cites other one-step families (shortcut, consistency and MeanFlow models; p. 3).

    • Comparing TD-Drift with at least one such one-step alternative trained with a TD objective would sharpen the contribution. So would an FOM-Direct baseline that backpropagates through the 10-step ODE solver, which the paper currently excludes (p. 6, ll. 321–322).

    • Reporting pre-training wall-clock time next to Fig. 7 would show the total cost.

3 comments

We thank the reviewer for their constructive feedback and clear re-evaluation criteria and we address main (W)eaknesses and (Q)uestions below.

Q1/W1: Frozen or fine-tuned baselines, normalization, pre-training steps and InFOM results comparison

In order to remain faithful to the original implementation, our implementation of InFOM (and flow-based pipelines) differed from drift-based pipelines in three aspects: (i) it pre-trained for fewer steps, (ii) it used a different normalization scheme, and (iii) it kept fine-tuning during policy extraction. As suggested, we have removed these confounders. We first matched pre-training steps and normalization:

DomainInFOM (matched steps and normalization)InFOM (original setting)
cheetah279.8 ± 9.4304.6 ± 13.3
quadruped406.8 ± 18.3458.4 ± 36.9
walker365.9 ± 9.6364.4 ± 7.2
jaco177.4 ± 2.6175.0 ± 2.5

InFOM’s performance plateaus early, and the method does not benefit from additional training steps, nor from a different normalization scheme. To further close the gap with the drifting-based pipeline, we can additionally freeze the occupancy model and the intention encoder during policy extraction. This is also a rational decision since the occupancy model does not depend on the extracted policy, and hence updating it is not required for policy improvement. We compare frozen and fine-tuned InFOM with matched steps and normalization.

DomainFinetunedFrozen
cheetah279.8 ± 9.4284.5 ± 25.9
quadruped406.8 ± 18.3425.6 ± 64.2
walker365.9 ± 9.6390.0 ± 8.8
jaco177.4 ± 2.6179.3 ± 1.5

The change in performance is not significant, and the two settings (flow- and drifting-based) are now directly comparable: both share the same pre-training budget, normalization, and freeze occupancies and intentions during policy extraction. Thus, the differences we report come from the model class. As suggested, we have repeated our evaluation in this setting and updated results accordingly. Finally, we note that the InFOM runtimes in Fig. 7 do not include occupancy-model updates.

Q1/W2: Fairness and InFOM results comparison

Regarding the differences from the per-task results published by InFOM (Table 4): our reruns use the original InFOM codebase with two deviations: the occupancy model is frozen during extraction, pre-training steps and normalization change, and the discount factor is re-tuned per domain. As shown above, the first three changes are on average harmless, and the latter is chosen to maximize average return, as displayed in the previous response.

Q2/W1: Tuning budget and seeds

The hyperparameters inherited from InFOM differ on a domain basis, and were thus likely tuned by the authors in a comparable setting, as shown in the responses above. These include latent dimension (>=2 values), KL weight (>= 4 values), expectile τ\tau (3 values), and BC regularization α\alpha (2 values). The drifting-based pipeline tunes its parameters over a comparable grid: KL weight (4 values) and BC regularization α\alpha (2 values). There are two additional hyperparameters: the kernel temperature (3 values) and M (2 values). The difference in latent dimension was an experimental oversight, which we have now corrected by fixing this hyperparameter to the InFOM default. We have updated results accordingly, and they have not changed significantly. The discount factor (2 values) is re-tuned for all methods. We thus believe that tuning budgets are comparable.

Configurations were selected per domain on the mean return over one task (as reported in l.840) using three seeds. Reported results used five seeds, which include these three. To remove the optimistic bias, we will re-evaluate each method using 5 held-out seeds. As this is computationally intensive, we will update the manuscript for the camera ready version.

Q3: Effect of intentions As suggested, we have rerun InDOM with τ=0.5\tau=0.5 at extraction on two representative environments.

DomainExpectile 0.9 (paper)Expectile 0.5
cheetah301.0 ± 13.7275.6 ± 12.6
quadruped677.0 ± 6.5674.6 ± 1.9

We find that replacing expectile regression with an MSE objective does not significantly change the performance, which suggests that optimistic aggregation is not a major contributor to InDOM’s performance with respect to DOM.

As suggested, we have also expanded the KL regularization weight grid in Figure 4; we find that the benefit of higher weights gradually plateaus.

Q4: Consistency of the practical estimator (Eq. 16)

We thank the reviewer for pointing this out. Eq. 16 is not a consistent estimator of the attraction field in Eq. 11 when the behavior policy is stochastic. We modified the assumptions in l.910 in the manuscript accordingly. In the new Appendix F, we derive this bias in closed form. It follows that mπβm^{\pi_\beta} is a fixed point of the practical objective if and only if this bias vanishes, which holds in particular for a deterministic behavior policy. However, the bias can be removed by sampling several next actions a1′,…,aK′∼π^β(⋅∣s′)a'_1,\dots,a'_K \sim\hat{\pi}_\beta(\cdot\mid s') from a BC policy which is already learned during pretraining. The numerator and the denominator of Eq.16 are then averaged separately before taking the ratio. This recovers the ratio of averages, and the bias vanishes as K→∞K\to\infty.

Q5: Isolating the role of one-step generation and FOM-Direct

Applying other one-step generative models to occupancy modeling is not straightforward as their standard training objectives rely on samples from their target distribution, and to the best of our knowledge, no temporal-difference variants have been formulated for them yet. We thus focus on showing that a class of one-step generative models is competitive with the multi-step models so far used for occupancy modeling, and we do not claim that drifting represents the optimal model class. Having established that one-step models are competitive, exploring other one-step frameworks is a promising direction, but we consider this beyond the scope of this paper, and we have added it as a direction for future work.

Following the reviewer’s suggestion, we also ran a FOM-Direct baseline that extracts the policy by backpropagating through the ODE solver:

DomainFOM-DirectFOMDOM-Direct
cheetah344.7 ± 12.0286.0 ± 12.4256.4 ± 46.0
quadruped539.0 ± 13.1538.2 ± 8.4662.3 ± 10.8
walker551.3 ± 7.7500.5 ± 8.2413.6 ± 12.9
jaco207.6 ± 2.9153.0 ± 1.5154.8 ± 3.4

Interestingly, FOM-Direct performs well, despite prior reports that gradients through iterative samplers are unstable and led to poor gradient estimates [1, 2]. However, it is slower per update than DOM-Direct, since each gradient step backpropagates through all integration steps. Finally, it remains below InDOM-Direct. Understanding why gradients through the flow sampler are informative in this setting, despite prior reports, is an interesting direction for future work.

W2: Scope

Our contribution is not only methodological, but also largely focuses on systematic empirical validation of design choices. Each variant and baseline we considered was thus extensively tuned. Considering computational constraints, we thus chose to prioritize a rigorous fine-grained evaluation over a broad sweep over environments.

W2/Q5: Efficiency is shown for extraction only

We focused on timing policy extraction, as the pre-trained occupancy model can be reused for multiple rewards, and its cost is amortized. Nevertheless, as suggested, we have added the computational cost of pre-training for drifting models (6.37ms per training step for InDOM) and compared it to that of flow-based models (2.23ms per training step for InFOM) in Appendix E.4. All experiments run on a single Nvidia A100. Drifting is thus significantly more compute-intensive and requires more gradient steps to converge. We believe that this is a fair cost to pay in exchange for generally faster policy extraction, which is only paid once since the pre-trained occupancy model can be reused for all rewards. On a side note, we found and corrected an error in our timing analysis for policy extractions, and updated it accordingly. While all drifting-based approaches offer significantly faster (>2x) policy extraction than their flow-based counterparts, InDOM-Direct requires larger forward passes and is 1.2x faster.

W2: Comparison to FB

FB is not a generative occupancy model and follows a different, zero-shot protocol. We include it as a reference point for occupancy-based methods in general, and its results (strong on cheetah and walker, weaker on quadruped and jaco) show that neither family dominates. Our claims concern the generative occupancy pipeline. We have revised the abstract to state the improvement relative to generative occupancy models explicitly.

W2: Comparison to FOM

We moved FOM results from Table 2 to Table 1 for completeness: while FOM performs well, it does not close the gap with the better drifting-based methods such as InDOM-Direct.

Minor Points

Thank you for the fine-grained feedback. We have updated the text in Fig. 4 and Table 2, and listed the additional hyperparameters in Table 3. We have sanity checked our implementation of FB, and the results are consistent with the literature [3, 4].


We thank the reviewer for the detailed and actionable feedback, which we have integrated into the manuscript. In particular, we believe that the new analysis of Bellman consistency and the controlled comparison between flow and drift models have strengthened our submission. We hope these additions address the reviewer’s concerns, and we would be grateful if they would consider updating their score. We are happy to answer any remaining questions.

[1] Zheng et al. “Intention-conditioned flow occupancy models.” ICLR 2026

[2] Park et al. “Flow q-learning.” ICML 2025

[3] Mohan et al. "Is Manipulation Hard for Successor Measures?" 3rd RLBrew Workshop at RLC 2026

[4] Jeen et al. "Zero-shot reinforcement learning from low quality data." NeurIPS 2024

Comments on the paper

1 comment

We would like to thank all reviewers for their informative and constructive feedback. In the following, we summarize the main changes that were suggested and applied to the revision; we marked them in blue for ease of parsing.

  • We have updated the main experimental protocol to use the same number of pre-training steps and the same normalization scheme across flow-based and drifting-based methods, and to keep occupancy models frozen during policy extraction. We additionally re-evaluated drifting-based methods to account for a mismatch in one hyperparameter, namely the latent dimension of intention embeddings. Results do not change significantly despite the streamlined setting: previous runs had converged, and most methods were not particularly sensitive to the latent dimension. Nevertheless, we believe that these changes contribute to a cleaner experimental setup. (pbpw, iNKR)
  • We expanded the study of conditional modeling in Section 5.2 to consider a wider range of KL regularization weights, finding that performance gains gradually plateau. (pbpw)
  • We recomputed and extended our study of computational costs in Section 5.4 and Appendix E.5 to account for an imprecise initial evaluation, and to include an analysis of pre-training costs. (pbpw, iNKR, LyUy)
  • We evaluated InDOM-Direct against an offline RL baseline on progressively smaller sets of reward-labeled data in Appendix E.3, finding that InDOM-Direct only requires a small amount of reward labels. (iNKR, LyUy)
  • We introduced an ablation over the expectile for policy extraction in Appendix E.4, finding that high expectiles do not explain the performance of InDOM. (pbpw)
  • We added a theoretical study of soundness and bias of TD-Drift objectives in Appendix F. (pbpw, iNKR)

We thank the reviewer for prompting each of these changes and extensions, which we believe strengthened our work. For further clarification, we refer to individual responses. We look forward to further discussions in the rebuttal phase.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.