acceptodds
Under review as a conference paper at ICLR 2027

T-MAC: Target-Motion Actor-Critic for Generative Reinforcement Learning

Abstract

Generative policies can represent complex and multimodal action distributions, which makes them attractive for continuous control. However, in online reinforcement learning, the target of such a policy is defined by a critic that changes after every update, so the policy keeps chasing a moving target. Existing methods train the policy toward where the target is now, but do not use how the target has moved. We propose Target-Motion Actor-Critic (T-MAC), a flow-based actor-critic method that learns from both signals. In addition to the current-target loss, T-MAC trains the cross-update change of the policy velocity field to follow the target motion. T-MAC estimates the target motion by comparing the Jordan–Kinderlehrer–Otto (JKO) responses of successive critics on the same source actions, and uses a response-magnitude gate to weight each state by how far its target moves. Experimental results on six MuJoCo and four HumanoidBench tasks show that, compared with representative model-free and generative-policy RL baselines, T-MAC achieves state-of-the-art performance with consistent improvements across all ten tasks.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, generative policy, continuous control

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_zjJsOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Target-Motion Actor-Critic, with a central idea around using the "trajectory" of a critic through its updates as a target (or as part of the target), rather than only the current target at a given iteration. The idea is elegant and largely well-motivated. Although the theory, algorithmic framework are certainly non-trivial, the idea is so intuitive that I am surprised it has not been considered before. This is a compliment, and I have questions about this below. After setting up this idea of target motion with appropriate metrics, loss functions, theoretical justification, and algorithm, the paper then proceeds to an evaluation on standard RL benchmarks. The empirical analysis of target motion is largely rigorous, and the ablations are sufficient.

Critical strengths
  • The paper is well organized and well written. The theory is relatively clean and clear to follow.
  • The central idea is intuitive and largely well-motivated. The ensuing empirical results are equally encouraging.
Critical weaknesses
  • The choice of flow-matching as the base (only) policy class is interesting but not sufficiently justified and, in my opinion, actually distracts from the central contribution.
Questions for authors
  1. Nothing wrong with flow matching, but this idea seems to potentially be a more general contribution to the RL literature. Target motion is used to induce a bias in the policy's velocity field; but could this idea work for any policy representation? How much would the base algorithm and loss functions need to be adapted? What is the justification for focusing on flow matching? This is perhaps especially relevant because the method is only evaluated on standard mujoco tasks (nothing wrong with this), for which the purported expressiveness of flow-matching and diffusive policies are perhaps not necessary.

  2. It is unclear how equation (4) would be used in online learning. Where do the target samples come from? (addendum: I am leaving this comment, but this is addressed a bit later. Perhaps mention this in Section 2.2 if the authors feel it helps readability).

  3. I suggest a light addition to the appendix that further describes Wasserstein grad. flows and JKO. 2.3 is a bit dense and is critical enough for the rest of the paper, that it is probably worth making things self-contained. Each of the terms can then be defined and so forth.

  4. I have to admit that the notion of target motion does not seem to be of first order importance. Yes, two different critic trajectories could arrive to the same place, but the reasons for this are myriad. One has nothing (or little) to do with the policy itself but rather the data (state trajectories) it experienced. (another addendum: this looks to be addressed in equation (10); again, the reader is sort of left thinking about this for quite a while, when reading this paper, before it is resolved).

  5. On the other hand, and without being terribly rigorous with this comment, something about this idea does not seem terribly novel. Of course many methods exist to deal with the stability of critic dynamics as it updates; I believe T-MAC is pretty different from these (which date all the way to DQN and probably earlier). What about PreQN (Achiam et al., 2019), which uses an NTK-based analysis of divergence in deep Q-learning and constrains each update to preserve the critics outputs on previously sampled data. Related are the work on Bellman error versus value error (Fujimoto et al., 2022) and the plasticity and capacity-loss literature (Kumar et al., 2021; Lyle et al., 2022), which characterize how much the critics function actually changes under successive updates. How is T-MAC different from these (and possibly others)?

  6. I am curious about the source distribution and assume this is used everywhere in both the contruction related to eqns (10)-(14) and the implementation. What is this source distribution and is this a bottleneck? I assume these data have to be "large" or else there are issues with generalization and so forth. NRCN_{RC} appears to set this; is the method sensitive to this hyperparameter?

  7. The if statement on line 6 of the pseudocode implies that lambdalambda is adaptive or decays. Is that correct? Or is this simply a way of "shutting off" target-motion?

  8. The target-motion analysis (sections 4.3 and protocol in the appendix) seem contrived. I fully appreciate that getting two trajectories to end up in the exact same critic is non-trivial; however, can the authors justify how this (if you will allow me to say "contrived") setup can lead to general conclusions.

4 comments
AuthorsOpenReview ↗

Response to Reviewer zjJs (1/2)

Thank you for the detailed and constructive comments. Below we address each point in turn.

W1 & Q1: Why flow matching, and does the idea generalize?

Target motion is a general principle, and flow matching is the representation in which it can be supervised most directly.

Any actor-critic policy is trained toward a target that moves with the critic. A flow policy is parameterized by an action-space velocity field, so its cross-update change δvθ=vθ−vk\delta v_\theta=v_\theta-v_k is itself an action-space vector field. The JKO response is exactly such a field and does not depend on flow time, so it supervises δvθ\delta v_\theta along the whole flow trajectory without any change of representation. A one-step flow policy also keeps inference cheap, since T-MAC generates an action with one midpoint integration step (two network evaluations).

The same principle extends to other policy classes, given a matching parameterization of the policy change. For a Gaussian policy, the change of its mean and covariance would be expressed as a displacement. For a diffusion policy, the change of its score field would be supervised. Each extension needs its own derivation of the change, so we present flow matching as the most direct case and keep these extensions for future work. On the benchmarks, T-MAC achieves a higher TAR than the Gaussian-policy baselines SAC, TD3 and PPO on all ten tasks, including the high-dimensional Humanoid-v4 and H1hand tasks.

Q2: Where do the target samples in Eq. (4) come from?

Thank you for the suggestion. We added a forward reference in Section 2.2. For each state, we draw Ncand=32N_{\mathrm{cand}}=32 actions from the frozen policy and take the one with the highest value under the updated critic, acur=arg⁡max⁡iQk+1(s,ai)a_{\mathrm{cur}}=\arg\max_i Q_{k+1}(s,a_i). The targets are generated online from the policy itself, and no expert data is needed.

Q3: Self-contained background on Wasserstein gradient flows and JKO

We added a short self-contained primer as Appendix F, and Section 2.3 points to it. The primer defines each term of the JKO step, including the W2W_2 proximity term, the critic potential and the entropy regularizer with its reference measure. It also gives the transport-map view of the response, explains why the response difference recovers ∇aδQk\nabla_a\delta Q_k to first order under a shared source (Theorem 1), and describes the particle approximation used in our implementation.

Q4: Target motion versus differences in the data

Thank you for this question. The shared source ensures that the teacher measures only the change between the two critics, whatever caused it.

Two critic histories can indeed differ because of the data each critic was trained on. Whatever moved the critic, the actor should follow the resulting motion of its target. Both proximal problems start from the same frozen source μk(⋅∣s)=πθk(⋅∣s)\mu_k(\cdot|s)=\pi_{\theta_k}(\cdot|s), with the same 64 particles and the same reference measure, so the response difference contains no contribution from a change in the sampled actions. With different sources, the difference would contain the additional drift term −α∇alog⁡(dμ~k/dμk)-\alpha\nabla_a\log(d\tilde\mu_k/d\mu_k) (Eq. 15). Following your comment, Section 3.2 now states this before Eq. (10).

Q5: Relation to PreQN, Bellman vs. value error, and plasticity

These works study the critic, and T-MAC turns the change of the critic into a supervision signal for the actor.

PreQN [1] constrains each update to preserve the outputs of the critic on previously sampled data. Fujimoto et al. [2] analyze the gap between Bellman error and value error. Kumar et al. [3] and Lyle et al. [4] characterize how the function and capacity of the critic change under successive updates. All of these study, constrain or regularize how the critic changes. T-MAC leaves critic training unchanged and uses the change between successive critics, evaluated from a shared source, as a supervision signal for the actor, so it can be combined with these critic-side methods. We added this discussion to Appendix A.2.

AuthorsOpenReview ↗

Response to Reviewer zjJs (2/2)

Q6: The source distribution and sensitivity to NsrcN_{\mathrm{src}}

The source is the current policy itself, not a dataset, and a small number of particles is enough.

The source distribution is the current policy πθk(⋅∣s)\pi_{\theta_k}(\cdot|s), frozen at the start of each policy update. The flow network, trained on replay-buffer states, handles generalization across states. The particles only provide a per-state Monte Carlo estimate of the response. The Humanoid-v4 results are listed below.

NsrcN_{\mathrm{src}}TARRelative to defaultTwo JKO solves / update
166968.2 ± 402.799.7%0.83 ms
327084.6 ± 375.9101.4%1.44 ms
64 (default)6988.8 ± 281.0100.0%3.50 ms

The three settings lie within 2% of each other, and 16 particles reach the same performance at about a quarter of the default JKO cost, so the source distribution is not a bottleneck.

Q7: The if statement in Algorithm 1

λ\lambda is fixed (λ=1\lambda=1) throughout training and is neither adaptive nor decayed. The condition only skipped the JKO computation when λ=0\lambda=0, which recovers current-target training. We removed it from the pseudocode and state this in Section 3.3.

Q8: How the matched-history test supports general conclusions

Thank you for this fair question. The matched-history test is indeed constructed on purpose as a controlled experiment, and the general conclusion comes from three results on real training runs.

In normal training, the history and the current critic always change together, so their effects cannot be told apart. Fixing the current critic isolates the effect of the history, which is why we construct the test this way. It checks a necessary condition. An update that ignores target motion must give the same result under two histories that share the current critic. Current-target training gives Dvel=0.0036D_{\mathrm{vel}}=0.0036, and T-MAC gives 0.27250.2725, so T-MAC responds to the history and current-target training does not. The conclusion that this signal improves learning rests on three results from real training runs.

  1. Paired checkpoints (Figure 3b). On checkpoints from actual training runs, one T-MAC update reduces the motion residual to 0.71±0.090.71\pm0.09 of the current-target update.
  2. Alignment (Table 2). On real checkpoints, the T-MAC teacher has the highest direction cosine (0.81±0.070.81\pm0.07) and the lowest relative error (0.69±0.100.69\pm0.10) with respect to the measured best-of-NcandN_{\mathrm{cand}} target displacement.
  3. Online controls (Appendix C.5). With the gate and per-state magnitude matched to T-MAC, a teacher whose direction depends only on the current critic falls below T-MAC on all three tested tasks, by up to 700.4 on Humanoid-v4.

References

[1] Achiam, Joshua, Ethan Knight, and Pieter Abbeel. "Towards characterizing divergence in deep Q-learning." arXiv preprint arXiv:1903.08894 (2019).

[2] Fujimoto, Scott, et al. "Why should I trust you, Bellman? The Bellman error is a poor replacement for value error." International Conference on Machine Learning (2022).

[3] Kumar, Aviral, et al. "Implicit under-parameterization inhibits data-efficient deep reinforcement learning." International Conference on Learning Representations (2021).

[4] Lyle, Clare, Mark Rowland, and Will Dabney. "Understanding and preventing capacity loss in reinforcement learning." International Conference on Learning Representations (2022).

Final Thanks

Thank you again for your encouraging and insightful review. Your suggestions on the exposition, the JKO background and the related work have made the paper clearer and more self-contained. We hope our responses address your questions and are happy to discuss further.

I thank the authors for their rebuttal. I already had a pretty high score and will leave it.

I do encourage the authors to think about my W1 and Q1. These are not actually weaknesses per se, but strengths and I encourage them to perhaps pursue this work as a general notion for all policy classes (or all actor-critic frameworks, etc).

Thank you for your positive assessment from the initial review and for maintaining your score. We also appreciate your encouragement to view target motion as a general idea for other policy classes and actor-critic frameworks.

Reviewer_uyKSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies online RL with flow-matching policies trained toward the best of their own samples under the latest critic, a target that moves with every critic update. T-MAC adds a loss asking the velocity field's change over one update to follow this motion, gated by how far the target moves. The motion is measured by moving the same samples one Wasserstein proximal (JKO) step under the old and new critics, and to first order it is the difference of their action gradients. T-MAC is reported best on all ten MuJoCo and HumanoidBench tasks.

Critical strengths
  1. The idea is simple and falsifiable. On Humanoid-v4 the gain over DIPO, the strongest baseline, is large, and shuffling the teacher across states removes most of it.

  2. The diagnostics are careful, with matched-history updates and shuffled, zero and direct-gradient teachers.

Critical weaknesses
  1. The paper does not show that target motion is what helps, rather than adding first-order critic information to a best-of-N target that uses no gradients. Theorem 1 and the direct-gradient ablation both indicate that the teacher is essentially ∇aQk+1−∇aQk\nabla_a Q_{k+1} - \nabla_a Q_k, which does not measure how the target moves. If the critic is only rescaled, Qk+1=(1+ϵ)QkQ_{k+1} = (1+\epsilon)Q_k, the target stays put, yet the teacher is about ϵ∇aQk\epsilon\nabla_a Q_k, the ordinary deterministic policy gradient (DPG) direction.

    The supplementary code also takes a single optimizer step per update, with the frozen snapshot θk\theta_k equal to the current parameters. The proximal part of the loss then contributes nothing, and "Zero teacher" cannot separate the teacher direction from the gated penalty. The missing control is a norm-matched current-gradient teacher, built from the new critic's action gradient alone or from its JKO response.

  2. The results do not support the abstract's claim of "consistent improvements across all ten tasks". By my rough calculation, a Welch t-test on table 1's five-seed results against the strongest baseline per task reaches p<0.05p < 0.05 only on Humanoid-v4, Swimmer-v4 and H1hand-crawl. The paper never says how TAR is computed, and on Humanoid-v4 T-MAC starts far above every baseline, which needs explaining. Several baselines look like failed runs (DACER on HumanoidStandup, QVPO on Walker2d). Most of the generative baselines also use a lower policy learning rate than T-MAC. I recommend to compare with the original papers' numbers and tune at least the learning rate.

Questions for authors
  1. Is the single optimizer step per update in the supplementary code the setting behind all reported results? In "Zero teacher", is the gate computed before or after the teacher is zeroed?

  2. Could you add the current-gradient teacher, norm-matched per state to the motion teacher, and the full JKO response under the new critic alone? I recommend to run them on Humanoid-v4 and at least two other tasks, with rows in table 2. This matters most for my score.

  3. How is TAR computed, and why do T-MAC and its variants start so far above the baselines on Humanoid-v4? Averages over the last tenth of training with confidence intervals would be more convincing.

  4. Were the baselines tuned, and did they use T-MAC's reward scale? What explains DACER on HumanoidStandup and QVPO on Walker2d?

6 comments
AuthorsOpenReview ↗

Response to Reviewer uyKS (1/3)

Thank you for the detailed and constructive comments. Below we address each point in turn.

W1 & Q2: Does the teacher carry information beyond the current critic?

A teacher whose direction comes from the updated critic alone performs worse than T-MAC on all three tasks and aligns worse with the target displacement, even when it has the same gate and per-state magnitude as T-MAC.

The rescaling case. Thank you for this sharp example, which helped us state an important property of the JKO teacher. As you note, the best-of-NcandN_{\mathrm{cand}} maximizer stays the same under Qk+1=cQkQ_{k+1}=cQ_k with c=1+ϵc=1+\epsilon, and to first order the teacher is along ϵ∇aQk\epsilon\nabla_aQ_k, the DPG direction. At a finite step, the JKO target also contains an entropy term, so rescaling the critic changes the proximal target [1].

12hW22(ν,μk)−cEν[Qk]+αKL(ν∥ϖs)=c{12chW22(ν,μk)−Eν[Qk]+αcKL(ν∥ϖs)}\frac{1}{2h}W_2^2(\nu,\mu_k)-c\mathbb E_\nu[Q_k]+\alpha\mathrm{KL}(\nu\Vert\varpi_s)=c\Big\lbrace \frac{1}{2ch}W_2^2(\nu,\mu_k)-\mathbb E_\nu[Q_k]+\frac{\alpha}{c}\mathrm{KL}(\nu\Vert\varpi_s)\Big\rbrace

This is a JKO step with step size chch and temperature α/c\alpha/c, so the teacher registers a change of critic scale as a change of the proximal target. Following your example, we added this as a remark after Proposition 1. To test whether this history information matters in practice, we ran the two controls you proposed. Everything except the teacher is unchanged (λ=1\lambda=1, Ncand=32N_{\mathrm{cand}}=32, Nsrc=64N_{\mathrm{src}}=64, budgets, and evaluation).

  • (i) Updated-critic JKO response uhQk+1u_h^{Q_{k+1}}, with the gate of T-MAC.
  • (ii) Norm-matched current gradient ∇aQk+1\nabla_aQ_{k+1}, with both the gate and the per-state magnitude of T-MAC. This is exactly the DPG direction you describe, and it differs from T-MAC only in the teacher direction.

The TAR results over five seeds are listed below (Section 4.3 and Appendix C.5).

TaskT-MAC(i) updated-critic JKO response(ii) norm-matched current gradient
Humanoid-v46988.8 ± 281.06470.5 ± 355.46288.4 ± 326.7
Ant-v45713.2 ± 83.75529.8 ± 94.65476.2 ± 107.3
H1hand-crawl839.9 ± 81.2788.6 ± 88.7770.3 ± 83.1

The alignment results in Table 2 are shown below.

TeacherDirection cosine ↑Relative error ↓
T-MAC teacher0.81 ± 0.070.69 ± 0.10
(i) updated-critic JKO response0.74 ± 0.060.78 ± 0.09
(ii) norm-matched current gradient0.70 ± 0.080.91 ± 0.14

T-MAC has the highest mean on all three tasks. Control (ii) shows the largest gap, with 700.4 on Humanoid-v4, 237.0 on Ant-v4 and 69.6 on H1hand-crawl. Because (ii) shares the gate and magnitude of T-MAC, the gain comes from the history-dependent direction of the teacher. This is the test you identified as most important, and it shows that first-order information of the current critic alone does not reproduce the gain.

W2 & Q1: Single optimizer step and the Zero-teacher ablation

All results use one actor step per update. Thank you for catching the issue with the Zero-teacher ablation. Your reading is correct, and we have revised its interpretation accordingly.

Single optimizer step. Yes, every reported result uses one actor optimizer step per policy update, starting from the frozen snapshot θk\theta_k (Appendix D.2). At θ=θk\theta=\theta_k we have δvθ=0\delta v_\theta=0, so the motion loss acts through its cross term

∇θLmot∣θk=−2hE[wkmot(s)(∂θvθk)⊤rkmot]\nabla_\theta\mathcal L_{\mathrm{mot}}\big\vert_{\theta_k}=-2h\mathbb E\big[w_k^{\mathrm{mot}}(s)(\partial_\theta v_{\theta_k})^\top r_k^{\mathrm{mot}}\big]

and works as a gated correction of the velocity field along the teacher. Under this one-step protocol, a single T-MAC update reduces the motion residual to 0.71±0.090.71\pm0.09 of the current-target update (Figure 3b).

Zero teacher. The gate is computed from the original JKO response before the teacher is set to zero. As you note, the remaining penalty then has zero gradient at θk\theta_k, so Zero teacher coincides with current-target training. Both reach 5936.7 ± 624.6 on Humanoid-v4, compared with 6988.8 ± 281.0 for T-MAC. Section 4.3 now presents this ablation as directional target-motion supervision versus current-target training, and control (ii) above tests the role of the teacher direction.

AuthorsOpenReview ↗

Response to Reviewer uyKS (2/3)

W3 & Q3: TAR, statistical evidence, and the Humanoid-v4 curves

TAR is the final-window average you suggest. Thank you for pointing out that the abstract should match the statistical evidence. Following your suggestion, we revised it to state that T-MAC achieves the highest mean return on all ten tasks, with a statistically significant aggregate improvement.

Definition of TAR. We report the total average return (TAR), which recent work uses on MuJoCo and HumanoidBench [2, 3, 4]. The TAR of a seed is the mean return over the evaluations in the final 10% of training, as in [5, 6]. This is the average over the last tenth of training that you recommend. DSAC-T [7] and DACER [8] use the same window but take its maximum, so our mean is a stricter choice. T-MAC is evaluated every T/20T/20 steps with 10 deterministic episodes, and the flow policy outputs actions directly without critic-based reranking. We report the mean ± std of TAR over five seeds (Section 4.1). Appendix C.6 adds 95% confidence intervals for all ten tasks, for example [6639.9, 7337.7] on Humanoid-v4 and [739.1, 940.7] on H1hand-crawl.

Statistical evidence. Thank you for running this check yourself. Your calculation agrees with ours. Against the strongest baseline on each task, Welch tests give p<0.001p<0.001 on Humanoid-v4, p=0.003p=0.003 on Swimmer-v4 and p=0.037p=0.037 on H1hand-crawl. To evaluate the claim across tasks, we follow the recommendation of rliable [9] for few-seed benchmarks. We normalize each T-MAC seed by the strongest baseline on the same task, gt,s=100(Rt,s/Rˉbaseline,t−1)g_{t,s}=100(R_{t,s}/\bar R_{\mathrm{baseline},t}-1), and compute the interquartile mean of the 50 scores. The aggregate gain is 11.48%, with a 95% hierarchical-bootstrap CI of [6.07%, 18.52%]. The bootstrap uses 200,000 replicates, and each replicate resamples tasks and then the T-MAC and baseline seeds within each task.

Humanoid-v4 curves. Thank you for spotting this. Your question led us to recheck the full pipeline, and we found two causes of the high starting point. (i) The plotting script extended each curve's first evaluation back to iteration 0. (ii) The T-MAC runs on Humanoid-v4 and H1hand-crawl, as well as the Humanoid-v4 ablation runs, had been resumed after a crash, and the restart overwrote their early evaluation logs. We reran all of these experiments from scratch with complete logs and regenerated Figures 2, 4 and 5. Every curve now starts at its first logged evaluation, and all methods start from similarly low returns. T-MAC remains the best method on both tasks (6988.8 ± 281.0 and 839.9 ± 81.2), and the ordering of every ablation is preserved.

AuthorsOpenReview ↗

Response to Reviewer uyKS (3/3)

W3 & Q4: Baseline configurations

Following your suggestion, we swept the actor learning rate of the two low-scoring baselines. Both improve with a larger learning rate, and both still stay far below the strongest methods.

Configurations. Each baseline uses the configuration recommended by its paper or official implementation. For example, DACER uses actor and critic learning rates of 10−410^{-4} [8]. Only the quantities that set the interaction budget are matched across methods. T-MAC uses a reward scale of 0.2, and each baseline keeps the reward preprocessing of its official implementation (Appendix D.4).

Learning-rate sweep. All other settings are kept fixed, and the TAR results over five seeds are listed below.

Method3×10−53\times10^{-5}1×10−41\times10^{-4} (official)3×10−43\times10^{-4}
DACER (HumanoidStandup-v4)49,738.6 ± 8,146.252,415.7 ± 16,996.362,587.4 ± 29,876.5
QVPO (Walker2d-v4)201.2 ± 83.2221.6 ± 195.8758.2 ± 503.4

A larger learning rate helps both methods, but they remain far below DIPO (149,635.6) and T-MAC (153,292.1) on HumanoidStandup-v4 and below SAC (3,966.0) and T-MAC (4,449.4) on Walker2d-v4. The ranking in Table 1 therefore stays the same.

Why these two runs are low. Your concern about failed runs led us to inspect their training logs, which show no NaN values or early termination. DACER reaches its entropy target early, and its return stays nearly flat from the first evaluation onward, which points to premature convergence. At a larger learning rate its seeds spread further apart, which is consistent with unstable gradients through its diffusion chain. QVPO is evaluated with its own action selection, which executes the best of 32 sampled actions under its critic as in its original paper [11]. Its advantage weighting gives zero weight to samples with negative advantage, so the walker recovers slowly after a fall. DPMD [10] reports similar weaknesses of both methods, and our DACER results are close to its independent reproduction.

References

[1] Jordan, Richard, David Kinderlehrer, and Felix Otto. "The variational formulation of the Fokker-Planck equation." SIAM Journal on Mathematical Analysis 29.1 (1998): 1-17.

[2] Zhan, Guojian, et al. "Bootstrap off-policy with world model." Advances in Neural Information Processing Systems 38 (2025).

[3] Wang, Likun, et al. "Off-policy reinforcement learning with model-based exploration augmentation." Advances in Neural Information Processing Systems 38 (2025).

[4] Zhang, Tianyi, et al. "Langevin rollout optimization for modelic reinforcement learning." Forty-third International Conference on Machine Learning (2026).

[5] Graesser, Laura, et al. "The state of sparse training in deep reinforcement learning." International Conference on Machine Learning. PMLR, 2022: 7766-7792.

[6] Martinez, Fernando, et al. "Stackelberg coupling of online representation learning and reinforcement learning." The Fourteenth International Conference on Learning Representations (2026).

[7] Duan, Jingliang, et al. "Distributional soft actor-critic with three refinements." IEEE Transactions on Pattern Analysis and Machine Intelligence 47.5 (2025): 3935-3946.

[8] Wang, Yinuo, et al. "Diffusion actor-critic with entropy regulator." Advances in Neural Information Processing Systems 37 (2024).

[9] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." Advances in Neural Information Processing Systems 34 (2021): 29304-29320.

[10] Ma, Haitong, et al. "Efficient online reinforcement learning for diffusion policy." Forty-second International Conference on Machine Learning (2025).

[11] Ding, Shutong, et al. "Diffusion-based reinforcement learning via Q-weighted variational policy optimization." Advances in Neural Information Processing Systems 37 (2024).

Final Thanks

Thank you again for your careful and rigorous review. Your suggestions directly improved the paper. The current-critic controls you proposed now give the most direct evidence for our main claim, and your questions on TAR, the Humanoid-v4 curves and the baselines made the evaluation more complete. In the revision, we added the current-critic controls to Section 4.3, Table 2 and Appendix C.5, corrected the interpretation of the Zero-teacher ablation, defined TAR and added confidence intervals and the aggregate test (Section 4.1 and Appendix C.6), revised the abstract, and reran the affected experiments. We hope these changes address your concerns, and we are happy to provide further details during the discussion.

Reviewer_es5eOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Target-Motion Actor-Critic (T-MAC), a flow-based online RL method that augments standard current-target policy learning with supervision for how the policy’s target moves across critic updates. The proposed T-MAC was evaluated using six MuJoCo tasks and four HumanoidBench tasks. T-MAC reports the competitive mean Total Average Return (TAR) among the compared baselines on all ten tasks.

Critical strengths

S1. T-MAC's use of the JKO scheme to construct the target-motion teacher is a genuinely novel application of Wasserstein gradient-flow geometry to the temporal dimension of critic updates, as opposed to the spatial dimension used by FlowRL (Lv et al., NeurIPS 2025) for regularization.

Critical weaknesses

W1. Missing comparison to GORL (ICML 2026) weakens the SOTA claim.

GORL (Zhang et al., ICML 2026, ref1) addresses the same problem (online RL with flow/diffusion policies for continuous control) on similar benchmarks (DMControl, humanoid tasks with 21-dimensional actions). Given that GORL is a published ICML 2026 paper on nearly the same problem, its absence from the related work and experiments is a significant omission that weakens the SOTA claim.

W2. The source of performance gains is not sufficiently isolated.

T-MAC's current-target branch samples NcandN_{cand} = 32 actions from πθk\pi_{\theta_k} and selects argmax Qk+1Q_{k+1}. No ablation isolates NcandN_{cand}.

W3. Computational cost is unreported.

T-MAC computes two JKO-style particle solves per policy update, each using NsrcN_{src} = 64 particles with up to 50 fixed-point iterations. In addition, it samples NcandN_{cand} = 32 candidate actions per state for the current-target branch. The experimental setup reports only environment-step budgets (1M or 5M steps) and hardware (8× RTX 5090), but not wall-clock training time, GPU-hours, per-iteration overhead, or time-to-reward. This is particularly notable in the context of GORL, which explicitly reports a cost–benefit analysis.

[Ref1] Zhang, C., Wan, Z., Chen, F., Yang, F., Feng, L., Zhou, Y., Yu, X., You, Y., Tsang, I. and An, B., Generative Online Reinforcement Learning. ICML 2026.

Questions for authors

Q1. Relation to GORL (ICML 2026). GORL is not discussed in the related work or compared in the experiments; I am wondering whether the two approaches are compatible, please add analysis in the related works. And it would be better to compare GORL with T-MAC in the same continuous-control tasks.

Q2. Attribution of gains (NcandN_{cand} and regularization ablation). Please provide an ablation over NcandN_{cand} in the current-target construction. Additionally, please compare against a current-target baseline augmented with a simple old-policy consistency term but without the JKO teacher. This is essential to determine whether the reported gains are primarily due to target-motion supervision or to best-of-NcandN_{cand} critic-guided action selection and implicit regularization.

Q3. Computational cost. Please report wall-clock training time, GPU-hours, and per-iteration overhead for T-MAC and the main baselines (at least FlowRL, DPMD, SAC, PPO) on at least 2–3 representative tasks. Specifically, what fraction of the additional training cost is due to (a) the two JKO particle solves (NsrcN_{src} = 64, ≤ 50 iterations each), and (b) the NcandN_{cand} = 32 candidate action sampling?

Q4. Robustness to critic noise. The target-motion teacher is formed from the difference between two learned neural critics, uhQk+1−uhQku^{Q_{k+1}}_h − u^{Q_k}_h. Please clarify how robust the motion branch is to critic-update noise.

5 comments
AuthorsOpenReview ↗

Response to Reviewer es5e (1/3)

Thank you for the detailed and constructive comments. Below we address each point in turn.

W1 & Q1: Relation to GoRL and direct comparison

GoRL and T-MAC stabilize generative policies from two different sides, and the two ideas can be combined.

Relation. GoRL [1] moves policy optimization into a latent space. A Gaussian latent policy is trained with PPO, and a conditional generative decoder, refined on a slower timescale, maps latents to actions. T-MAC keeps the flow policy in action space inside an off-policy actor-critic loop and adds a new supervision signal, which is how the critic-defined target moves between critic updates. So GoRL changes where the policy is optimized, and T-MAC changes what supervises it. The decoder of GoRL is also trained toward targets that change from stage to stage, so supervising how the decoder follows these changes is a natural way to combine the two methods. This discussion is now in Appendix A.1.

Setup. We run the official GoRL implementation with its flow-matching decoder, GoRL(FM), and add rollout interfaces for Gymnasium MuJoCo and HumanoidBench. We first checked our use of the code in the original setting of [1]. On CheetahRun with the default 180M-step configuration, GoRL(FM) reaches a final return of 910.8 ± 5.2 over five seeds, slightly above the reported 883.4 ± 19.9. The official humanoid setup uses a Brax teacher, which is not available in Gymnasium MuJoCo or HumanoidBench, so we train GoRL from scratch with the latent dimension equal to the action dimension. All other hyperparameters follow the official defaults. Only the budget-dependent quantities are scaled to our 1M and 5M budgets, and the stage lengths keep their default ratios (Appendix D.4). GoRL is evaluated every T/20T/20 steps as in its official code, and its TAR is computed with the same script as for all other methods.

Results. GoRL is designed for large-scale parallel on-policy training, and its default configuration uses 180M environment steps. Our benchmark fixes the budget at 1M or 5M steps, where an off-policy method gains the most from reusing every transition. The TAR results over five seeds under these budgets are listed below (Table 1 and Figure 2).

TaskGoRL(FM)T-MAC
Humanoid-v42022.7 ± 1376.76988.8 ± 281.0
Ant-v41745.2 ± 200.25713.2 ± 83.7
HalfCheetah-v43679.0 ± 258.611571.8 ± 706.4
Walker2d-v4849.5 ± 437.74449.4 ± 873.2
Swimmer-v445.0 ± 3.0158.4 ± 12.1
HumanoidStandup-v4128040.5 ± 22792.5153292.1 ± 4557.8
H1-maze180.5 ± 35.1299.0 ± 15.5
H1-reach2879.8 ± 279.17417.7 ± 735.5
H1hand-reach1623.9 ± 282.26583.8 ± 1069.7
H1hand-crawl284.6 ± 51.1839.9 ± 81.2

Under these budgets, T-MAC reaches a higher TAR than GoRL on all ten tasks. The gap mainly reflects sample efficiency at small budgets, and the stability-oriented design of GoRL remains attractive when large-scale parallel simulation is available.

AuthorsOpenReview ↗

Response to Reviewer es5e (2/3)

W2 & Q2: Attribution of the gains

Target-motion supervision adds a large gain at every candidate count, and a generic old-policy consistency term recovers only a small part of this gain.

Candidate number. We vary Ncand∈{1,4,8,16,32,64}N_{\mathrm{cand}}\in\lbrace 1,4,8,16,32,64\rbrace on Humanoid-v4 with the motion branch off (λ=0\lambda=0, current-target training) and on (λ=1\lambda=1, T-MAC). Update times are synchronized measurements per policy update, and the results are listed below (Appendix C.4, Table 3).

NcandN_{\mathrm{cand}}λ=0\lambda=0λ=1\lambda=1 (T-MAC)GainT-MAC update time
14658.0 ± 318.05908.0 ± 387.0+1250.015.54 ms
44836.2 ± 329.56274.8 ± 382.6+1438.616.34 ms
85087.0 ± 342.06661.0 ± 401.0+1574.016.86 ms
165547.6 ± 478.46855.2 ± 330.1+1307.618.84 ms
32 (default)5936.7 ± 624.66988.8 ± 281.0+1052.121.40 ms
646084.9 ± 661.77059.6 ± 297.4+974.728.08 ms

The ablation you asked for separates the two effects. More candidates improve current-target training, and target motion adds a further 974.7 to 1574.0 on top of it at every candidate count. T-MAC with a single candidate (5908.0) already matches current-target training with 32 candidates (5936.7). We use 32 candidates by default, because moving to 64 raises the TAR by 1.0% and the update time by 31.2%.

Old-policy consistency. To make the control you proposed effective in our one-step setting, we anchor the penalty one update earlier, because a penalty toward θk\theta_k itself has zero gradient at θk\theta_k. We call this lagged-policy consistency. It keeps the same best-of-32 objective and samples and uses no JKO teacher or gate.

LB1=Lcur+βE∥vθ−stopgrad(vθk−1)∥22.\mathcal L_{\mathrm{B1}}=\mathcal L_{\mathrm{cur}}+\beta\mathbb E\big\lVert v_\theta-\mathrm{stopgrad}(v_{\theta_{k-1}})\big\rVert_2^2.

We set β\beta from gradient norms measured at early, middle and late checkpoints (100 batches each). With β=100\beta=100, the gradient of the consistency term matches that of the T-MAC motion loss in norm, and β=10\beta=10 is a weaker setting. The Humanoid-v4 results over five seeds are listed below.

MethodTAR
Current-target training5936.7 ± 624.6
+ lagged-policy consistency, β=10\beta=106254.6 ± 487.3
+ lagged-policy consistency, β=100\beta=1006072.8 ± 571.6
T-MAC6988.8 ± 281.0

The better consistency setting recovers 30.2% of the T-MAC gain and stays 734.2 below T-MAC, and a stronger penalty helps less. Your control therefore shows that what T-MAC adds is the direction of the change, which comes from the motion of the critic-induced target.

AuthorsOpenReview ↗

Response to Reviewer es5e (3/3)

W3 & Q3: Computational cost

With the cost-benefit protocol of GoRL, T-MAC uses 1.08× the wall-clock time of PPO and reaches 4.31× its TAR. It also reaches the final return of the strongest baseline using 55% to 65% of the training budget.

Per-update profile. We time one policy update on one RTX 5090 with Ncand=32N_{\mathrm{cand}}=32 and Nsrc=64N_{\mathrm{src}}=64 (Appendix D.6, Figure 6a). Of the additional cost you asked about, (a) the two JKO solves take 3.5 ms (16.4%) and (b) candidate generation and critic scoring take 6.8 ms (31.8%). The flow-actor update takes 8.4 ms (39.3%) and other operations take 2.7 ms (12.6%), for a total of 21.4 ms.

End-to-end cost. Each run uses 5M transitions and one RTX 5090. We report the mean ± std over five seeds, sum GPU-hours over the seeds, and normalize time and TAR to PPO within each task as in GoRL [1] (Table 8).

MethodHumanoid-v4 h/seedGPU-hTimeTARH1hand-crawl h/seedGPU-hTimeTAR
T-MAC4.42 ± 0.1222.101.10×6.03×8.14 ± 0.2340.701.06×2.59×
FlowRL [2]4.31 ± 0.1121.551.07×4.87×8.02 ± 0.2140.101.04×2.15×
DPMD [3]4.70 ± 0.1523.501.17×4.61×8.47 ± 0.2642.351.10×1.38×
SAC4.08 ± 0.1020.401.02×3.79×7.78 ± 0.1938.901.01×2.00×
PPO4.01 ± 0.0920.051.00×1.00×7.70 ± 0.1838.501.00×1.00×

Averaged over the two tasks, T-MAC uses 1.08× the time of PPO and reaches 4.31× its TAR, compared with 1.06× and 3.51× for FlowRL, 1.14× and 3.00× for DPMD, and 1.01× and 2.90× for SAC (Figure 6b). The per-update overhead stays small end to end because policy updates run on the GPU while the CPU steps the environments, and 20.1 of the 21.4 ms overlaps with the 28.9 ms rollout interval on Humanoid-v4.

Time-to-reward. We measure when the five-seed mean evaluation curve first reaches the final TAR of the strongest baseline, and convert this to wall-clock time by scaling the measured full-run time.

TaskTargetT-MACFlowRL
Humanoid-v46090.2 (DIPO)3.25M steps (65%), ≈2.87 hnot reached
H1hand-crawl696.7 (TD3)2.75M steps (55%), ≈4.48 h4.25M steps (85%), ≈6.82 h

Q4: Robustness to critic-update noise

The JKO teacher is robust to critic noise, and its direction stays well aligned with the noise-free teacher at every tested noise level.

Noise that does not depend on the action leaves ∇aQ\nabla_aQ unchanged, so we perturb the parameters of the updated critic and keep the previous critic fixed. We calibrate the perturbation so that g=∇a(Qk+1−Qk)g=\nabla_a(Q_{k+1}-Q_k) has relative RMS perturbation rr. We then recompute the full JKO teacher with the source particles, policy snapshot and solver settings held fixed, over 20 checkpoints and 10 perturbations per level.

rrTeacher cosineRelative teacher error
0.100.993 ± 0.0040.13 ± 0.02
0.250.960 ± 0.0140.30 ± 0.04
0.500.850 ± 0.0400.56 ± 0.08

Even when the perturbation is half the size of the critic-change signal (r=0.50r=0.50), the teacher keeps a cosine of 0.850. Action-independent critic shifts also cancel exactly (Proposition 1), and the shared source removes the source-drift term (Eq. 15).

References

[1] Zhang, Chubin, et al. "Generative online reinforcement learning." Forty-third International Conference on Machine Learning (2026).

[2] Lyu, Lei, et al. "Flow-based policy for online reinforcement learning." Advances in Neural Information Processing Systems 38 (2025).

[3] Ma, Haitong, et al. "Efficient online reinforcement learning for diffusion policy." Forty-second International Conference on Machine Learning (2025).

Final Thanks

Thank you again for your time and constructive suggestions. These suggestions directly strengthened the paper. The GoRL comparison, the attribution ablations and the cost analysis now make the evaluation more complete. In the revision, we added GoRL to the related work (Appendix A.1), Table 1, Figure 2 and Section 4.1, extended the NcandN_{\mathrm{cand}} ablation (Appendix C.4), and added the computational-cost analysis with the cost-return comparison and time-to-reward (Appendix D.6). We hope our responses address your concerns, and we are happy to provide further details during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.