- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces RLDT, a new online reinforcement learning algorithm for fine-tuning flow-matching policies. Specifically, RLDT applies the SVGD in Soft Q-Learning to a multi-step flow strategy and provides a method for assigning SVGD update directions to each denoising step. The final training objective consists of three components: RL loss, consistency loss, and constraint loss. The experiment results indicate that RLDT achieves or outperforms the baseline method on three benchmarks OpenAI Gym, Furniture-Bench and Robomimic.
Critical strengths
The description of the methodology section in this paper is relatively detailed, the overall process is clear, and easy to understand.
Critical weaknesses
-
In my opinion, the proposed RLDT mainly adapts the the SVGD-based policy update of Soft Q-Learning (Eq. (11), following Wang & Liu, 2016) to flow policies, so the overall novelty is quite limited.
-
Some information in Section 4 are not clearly described. Please refer to the Questions for Authors for details.
Questions for authors
-
In Figure 2, RLDT shows a clear performance drop at around 3M steps on Hopper and Walker2d tasks, and then recovers rapidly from the lowest reward. What is the reason for this phenomenon? Why does the similar issues not occurr on other tasks?
-
According to Table 3, when , the entire training crashes with 0 success rate. Could the authors explain the reason? In addition, the results on the task Lamp-Low are quite sensitive to and at the early training stage (45M). How did author choose these parameters? At the late training stage (215M), all setting achieve almost the same results. In particular, the setting (row 4) has the same final success rate as the default setting (row1). Does these results suggest that the consistency loss can be removed when training runs long enough? The authors are also encouraged to add more ablation results on other benchmarks to demonstrate the effectiveness of each component.
2 comments
In Figure 2, RLDT shows a clear performance drop at around 3M steps on Hopper and Walker2d tasks, and then recovers rapidly from the lowest reward. What is the reason for this phenomenon? Why does the similar issues not occurr on other tasks?
We have run additional experiments to study the causes behind the dip observed in the Hopper and Walker2d. Our analysis indicates that the critic update is the primary underlying factor. Our Gym experiments used a one-step target (), whereas FurnitureBench and Robomimic used a -return with . With , reward information propagates one step per update, so during the initial training iterations the critic’s action gradient is poorly informed and the actor follows it into a lower-reward region before the critic catches up. Setting on Gym, as in the other benchmarks, removes the dip (Figure 2). No other hyperparameter was changed. We have used uniformly across all benchmarks in the revision.
According to Table 3, when , the entire training crashes with 0 success rate. Could the authors explain the reason?
At the beginning of training, the critic cannot provide accurate information of Q gradient in sparse reward setting. If the constraint loss is removed, the policy will change drastically and exploit the wrong Q, leading to worse performance. Since we use sparse rewards, once the actor collapses to 0 success rate, it no longer provides useful information for the critic and it stays at 0.
How did author choose these parameters?
We tune the parameter on the lamp-low task and choose the best for the remaining tasks. We use the same setting for all the tasks. This suggests the RLDT is robust to the parameter setting. We find that tuning parameter could lead to only minor performance improvement for certain tasks (Fig. 9).
Does these results suggest that the consistency loss can be removed when training runs long enough?
Consistency loss only helps with learning efficiency. This is because the SVGD field does not require the particles to be close to the sampling distribution. As is shown in Fig. 9, non-zero consistency loss weight only helps for faster learning for the Lamp-low task, when the initial success rate is low. For the Robomimic Square and Transport task , given a better initial success rate, removing consistency loss does not influence the learning. We have clarified this in the manuscript (LL 967).
As the reviewer correctly points out, our work builds on SQL[Haarnoja et al. (2017)]. Similarly to the latter, our method amortizes the SVGD field into a network. However, the two methods differ in the way that they define the SVGD field. SQL uses a one-step sampler a=fϕ(ξ;s) whose output is the action, so the transport field can be applied to the output directly. This does not apply to a flow policy, where a network outputs a velocity at each of T steps and the action is obtained only by integration. This creates two problems that do not arise for one-step samplers that predict actions:
(1) Supervising every denoising step through one-step prediction a† . We project each intermediate state onto the action manifold through the one-step prediction a†=aτ+(1−τ)vθ. This lets us evaluate the SVGD transport field at every step, avoiding both back-propagation through the sampler and log-likelihood estimation.
(2) Projecting an action update into a network update. The SVGD displacement ϵϕ(a†) lives in action space, but the network output v is related to the approximate action after denoising a† through a map a†=g(v), where g is the above-mentioned one-step prediction via Euler integration, with Jacobian J=∂a†/∂v. Treating v as a free variable, there are two ways to map the update back to the network output.
Gradient formulation (naive application of SVGD in SQL). Set ∂Jπ/∂a†∝ϕ and apply the chain rule, we get Δv=ϵJ⊤ϕ. The realized displacement is then
a~†−a†≈JΔv=ϵJJ⊤ϕ(a†)
which is a distorted field not equivalent to the required SVGD field ϵϕ .
Transport formulation (ours). Differentiating a†=g(v) gives Δa†=JΔv to first order. We then set the transport displacement Δa†=ϵϕ(a†), by definition of T=id+ϵϕ, to yield JΔv=ϵϕ(a†). We can then solve for Δv:
Δv=ϵJ+ϕ(a†)=ϵJ⊤(JJ⊤)−1ϕ(a†).
The realized displacement is now consistent with the SVGD field by design, since a~†−a†≈JΔv=ϵϕ.
The gradient formulation operates on a distorted field, which leads to noise in the update signals and therefore slower learning. We provide a more extensive analysis and empirical comparisons in Fig. 6, where we show that using the SVGD update empirically leads to faster learning. Specifically, given the same 225M environment steps in one-leg med and lamp med settings, Transport formulation achieves 63.7% (vs. Gradient 31.7%) and 38.5% success rate (vs. Gradient 6.2%). For comparable performance, the Transport formulation uses only 99M environment steps, less than half of the Gradient formulation.
Overall, our method is novel because of two main technical innovations: (1) training multiple-step flow-matching policies without restrictions on the policy design; and (2) ensuring the network’s updates are consistent with the SVGD field by design, removing the distortions introduced by a SQL-style application of SVGD to flow-matching policies. We have clarified the novelty of our technical contribution in the manuscript’s introduction (Line 77, 86) and related work (Line 134), and experiments (Line 499).