- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Target-Motion Actor-Critic, with a central idea around using the "trajectory" of a critic through its updates as a target (or as part of the target), rather than only the current target at a given iteration. The idea is elegant and largely well-motivated. Although the theory, algorithmic framework are certainly non-trivial, the idea is so intuitive that I am surprised it has not been considered before. This is a compliment, and I have questions about this below. After setting up this idea of target motion with appropriate metrics, loss functions, theoretical justification, and algorithm, the paper then proceeds to an evaluation on standard RL benchmarks. The empirical analysis of target motion is largely rigorous, and the ablations are sufficient.
Critical strengths
- The paper is well organized and well written. The theory is relatively clean and clear to follow.
- The central idea is intuitive and largely well-motivated. The ensuing empirical results are equally encouraging.
Critical weaknesses
- The choice of flow-matching as the base (only) policy class is interesting but not sufficiently justified and, in my opinion, actually distracts from the central contribution.
Questions for authors
-
Nothing wrong with flow matching, but this idea seems to potentially be a more general contribution to the RL literature. Target motion is used to induce a bias in the policy's velocity field; but could this idea work for any policy representation? How much would the base algorithm and loss functions need to be adapted? What is the justification for focusing on flow matching? This is perhaps especially relevant because the method is only evaluated on standard mujoco tasks (nothing wrong with this), for which the purported expressiveness of flow-matching and diffusive policies are perhaps not necessary.
-
It is unclear how equation (4) would be used in online learning. Where do the target samples come from? (addendum: I am leaving this comment, but this is addressed a bit later. Perhaps mention this in Section 2.2 if the authors feel it helps readability).
-
I suggest a light addition to the appendix that further describes Wasserstein grad. flows and JKO. 2.3 is a bit dense and is critical enough for the rest of the paper, that it is probably worth making things self-contained. Each of the terms can then be defined and so forth.
-
I have to admit that the notion of target motion does not seem to be of first order importance. Yes, two different critic trajectories could arrive to the same place, but the reasons for this are myriad. One has nothing (or little) to do with the policy itself but rather the data (state trajectories) it experienced. (another addendum: this looks to be addressed in equation (10); again, the reader is sort of left thinking about this for quite a while, when reading this paper, before it is resolved).
-
On the other hand, and without being terribly rigorous with this comment, something about this idea does not seem terribly novel. Of course many methods exist to deal with the stability of critic dynamics as it updates; I believe T-MAC is pretty different from these (which date all the way to DQN and probably earlier). What about PreQN (Achiam et al., 2019), which uses an NTK-based analysis of divergence in deep Q-learning and constrains each update to preserve the critics outputs on previously sampled data. Related are the work on Bellman error versus value error (Fujimoto et al., 2022) and the plasticity and capacity-loss literature (Kumar et al., 2021; Lyle et al., 2022), which characterize how much the critics function actually changes under successive updates. How is T-MAC different from these (and possibly others)?
-
I am curious about the source distribution and assume this is used everywhere in both the contruction related to eqns (10)-(14) and the implementation. What is this source distribution and is this a bottleneck? I assume these data have to be "large" or else there are issues with generalization and so forth. appears to set this; is the method sensitive to this hyperparameter?
-
The if statement on line 6 of the pseudocode implies that is adaptive or decays. Is that correct? Or is this simply a way of "shutting off" target-motion?
-
The target-motion analysis (sections 4.3 and protocol in the appendix) seem contrived. I fully appreciate that getting two trajectories to end up in the exact same critic is non-trivial; however, can the authors justify how this (if you will allow me to say "contrived") setup can lead to general conclusions.
4 comments
Response to Reviewer zjJs (2/2)
Q6: The source distribution and sensitivity to
The source is the current policy itself, not a dataset, and a small number of particles is enough.
The source distribution is the current policy , frozen at the start of each policy update. The flow network, trained on replay-buffer states, handles generalization across states. The particles only provide a per-state Monte Carlo estimate of the response. The Humanoid-v4 results are listed below.
| TAR | Relative to default | Two JKO solves / update | |
|---|---|---|---|
| 16 | 6968.2 ± 402.7 | 99.7% | 0.83 ms |
| 32 | 7084.6 ± 375.9 | 101.4% | 1.44 ms |
| 64 (default) | 6988.8 ± 281.0 | 100.0% | 3.50 ms |
The three settings lie within 2% of each other, and 16 particles reach the same performance at about a quarter of the default JKO cost, so the source distribution is not a bottleneck.
Q7: The if statement in Algorithm 1
is fixed () throughout training and is neither adaptive nor decayed. The condition only skipped the JKO computation when , which recovers current-target training. We removed it from the pseudocode and state this in Section 3.3.
Q8: How the matched-history test supports general conclusions
Thank you for this fair question. The matched-history test is indeed constructed on purpose as a controlled experiment, and the general conclusion comes from three results on real training runs.
In normal training, the history and the current critic always change together, so their effects cannot be told apart. Fixing the current critic isolates the effect of the history, which is why we construct the test this way. It checks a necessary condition. An update that ignores target motion must give the same result under two histories that share the current critic. Current-target training gives , and T-MAC gives , so T-MAC responds to the history and current-target training does not. The conclusion that this signal improves learning rests on three results from real training runs.
- Paired checkpoints (Figure 3b). On checkpoints from actual training runs, one T-MAC update reduces the motion residual to of the current-target update.
- Alignment (Table 2). On real checkpoints, the T-MAC teacher has the highest direction cosine () and the lowest relative error () with respect to the measured best-of- target displacement.
- Online controls (Appendix C.5). With the gate and per-state magnitude matched to T-MAC, a teacher whose direction depends only on the current critic falls below T-MAC on all three tested tasks, by up to 700.4 on Humanoid-v4.
References
[1] Achiam, Joshua, Ethan Knight, and Pieter Abbeel. "Towards characterizing divergence in deep Q-learning." arXiv preprint arXiv:1903.08894 (2019).
[2] Fujimoto, Scott, et al. "Why should I trust you, Bellman? The Bellman error is a poor replacement for value error." International Conference on Machine Learning (2022).
[3] Kumar, Aviral, et al. "Implicit under-parameterization inhibits data-efficient deep reinforcement learning." International Conference on Learning Representations (2021).
[4] Lyle, Clare, Mark Rowland, and Will Dabney. "Understanding and preventing capacity loss in reinforcement learning." International Conference on Learning Representations (2022).
Final Thanks
Thank you again for your encouraging and insightful review. Your suggestions on the exposition, the JKO background and the related work have made the paper clearer and more self-contained. We hope our responses address your questions and are happy to discuss further.
I thank the authors for their rebuttal. I already had a pretty high score and will leave it.
I do encourage the authors to think about my W1 and Q1. These are not actually weaknesses per se, but strengths and I encourage them to perhaps pursue this work as a general notion for all policy classes (or all actor-critic frameworks, etc).
Thank you for your positive assessment from the initial review and for maintaining your score. We also appreciate your encouragement to view target motion as a general idea for other policy classes and actor-critic frameworks.
Response to Reviewer zjJs (1/2)
Thank you for the detailed and constructive comments. Below we address each point in turn.
W1 & Q1: Why flow matching, and does the idea generalize?
Target motion is a general principle, and flow matching is the representation in which it can be supervised most directly.
Any actor-critic policy is trained toward a target that moves with the critic. A flow policy is parameterized by an action-space velocity field, so its cross-update change δvθ=vθ−vk is itself an action-space vector field. The JKO response is exactly such a field and does not depend on flow time, so it supervises δvθ along the whole flow trajectory without any change of representation. A one-step flow policy also keeps inference cheap, since T-MAC generates an action with one midpoint integration step (two network evaluations).
The same principle extends to other policy classes, given a matching parameterization of the policy change. For a Gaussian policy, the change of its mean and covariance would be expressed as a displacement. For a diffusion policy, the change of its score field would be supervised. Each extension needs its own derivation of the change, so we present flow matching as the most direct case and keep these extensions for future work. On the benchmarks, T-MAC achieves a higher TAR than the Gaussian-policy baselines SAC, TD3 and PPO on all ten tasks, including the high-dimensional Humanoid-v4 and H1hand tasks.
Q2: Where do the target samples in Eq. (4) come from?
Thank you for the suggestion. We added a forward reference in Section 2.2. For each state, we draw Ncand=32 actions from the frozen policy and take the one with the highest value under the updated critic, acur=argmaxiQk+1(s,ai). The targets are generated online from the policy itself, and no expert data is needed.
Q3: Self-contained background on Wasserstein gradient flows and JKO
We added a short self-contained primer as Appendix F, and Section 2.3 points to it. The primer defines each term of the JKO step, including the W2 proximity term, the critic potential and the entropy regularizer with its reference measure. It also gives the transport-map view of the response, explains why the response difference recovers ∇aδQk to first order under a shared source (Theorem 1), and describes the particle approximation used in our implementation.
Q4: Target motion versus differences in the data
Thank you for this question. The shared source ensures that the teacher measures only the change between the two critics, whatever caused it.
Two critic histories can indeed differ because of the data each critic was trained on. Whatever moved the critic, the actor should follow the resulting motion of its target. Both proximal problems start from the same frozen source μk(⋅∣s)=πθk(⋅∣s), with the same 64 particles and the same reference measure, so the response difference contains no contribution from a change in the sampled actions. With different sources, the difference would contain the additional drift term −α∇alog(dμ~k/dμk) (Eq. 15). Following your comment, Section 3.2 now states this before Eq. (10).
Q5: Relation to PreQN, Bellman vs. value error, and plasticity
These works study the critic, and T-MAC turns the change of the critic into a supervision signal for the actor.
PreQN [1] constrains each update to preserve the outputs of the critic on previously sampled data. Fujimoto et al. [2] analyze the gap between Bellman error and value error. Kumar et al. [3] and Lyle et al. [4] characterize how the function and capacity of the critic change under successive updates. All of these study, constrain or regularize how the critic changes. T-MAC leaves critic training unchanged and uses the change between successive critics, evaluated from a shared source, as a supervision signal for the actor, so it can be combined with these critic-side methods. We added this discussion to Appendix A.2.