acceptodds
Under review as a conference paper at ICLR 2027

Simple Flow Gradient: Fine-Tuning Flow Policies with the Direct Q-Gradient

11 views

Abstract

Generative policies trained by imitation rarely reach the reliability that deployment demands, and reinforcement learning (RL) offers a way to improve them through feedback from their own interactions. Off-policy methods reuse past experience for sample efficiency, but applying them to a generative policy raises two concerns identified by prior work: backpropagating the critic's gradient through the sampling chain may be unstable, and an update without an anchor to the data may exploit errors in the critic's value estimates. This work examines whether the direct Q-gradient, the most elementary such update, can fine-tune a pretrained generative policy without an anchor. Analyzing direct Q-gradient updates on a one-step flow policy, we find that a single sufficiently large update can erase pretrained competence. This motivates bounding the action space change of each episode's update while allowing continued adaptation beyond the pretrained policy. Simple Flow Gradient (SFG) applies this bound by projection to a pretrained flow policy updated with the direct Q-gradient; the bound is its only addition to the actor-critic recipe, which keeps no anchor term, distillation stage, or auxiliary network. Starting from converged behavior cloning policies under matched interaction budgets, SFG reaches a mean task completion score of 0.98 on five manipulation tasks from Robomimic and Franka-Kitchen, and a score of 0.8 more than twice as fast as the fastest baseline. Additional experiments show that SFG also applies to multistep flow policies.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, policy gradient, generative policies

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VoEhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies direct Q-gradient fine-tuning of pretrained flow policies without an actor loss that anchors the policy to demonstrations or its pretrained parameters. It observes that large updates can destroy pretrained performance and proposes Simple Flow Gradient (SFG), which limits the change in generated actions between successive committed policies. The constraint is measured on a fixed set of state–noise pairs and enforced by rescaling the accumulated parameter update at each episode boundary. The method combines this procedure with demonstration replay and an ensemble critic. Experiments primarily evaluate one-step MeanFlow policies, with additional results on multi-step policies and a point-cloud manipulation task.

Critical strengths

The paper examines a practically relevant problem: RL fine-tuning can initially destroy the capabilities acquired through imitation learning. The frozen-critic experiment and the measurements of action changes provide useful evidence that update magnitude deserves attention, even when the policy uses a one-step generator.

The proposed modification is relatively simple and produces promising results in the main comparison. I also appreciate that the paper reports settings where the constraint slows early learning and where the one-step method fails. These results help identify the limits of the approach rather than presenting only favorable outcomes.

Critical weaknesses

(a) The experimental coverage is too limited for me to assess how broadly the method is useful and how large its benefits are likely to be. The main comparison averages over five task settings, three of which are variants of Franka-Kitchen, while the component ablations cover only two tasks. The additional transport and point-cloud experiments are useful, but they do not establish consistent gains across a sufficiently diverse set of conditions. In particular, the one-step method fails on transport, and the multi-step results involve different pretrained policies and additional implementation considerations. Broader evaluation on further Robomimic tasks and suitable OGBench tasks would help determine whether the findings extend beyond the current settings. This matters because the reported benefit appears to depend on the competence of the initial policy and the task: preserving an already useful policy and learning from a weak initialization may require different update behavior.

(b) The paper motivates controlling policy changes, but does not convincingly establish why the proposed functional-step projection is the appropriate way to do so, or why it should be preferred over other constraints. The learning-rate ablations are helpful, but they do not resolve this question. Comparisons with an action-distance penalty relative to the previous policy, another trust-region formulation, or adaptive step-size control would be more informative about the specific contribution. These comparisons should use the same pretrained actor, critic, replay scheme, and moving reference policy. As it stands, the strongest controlled result is that the constraint avoids an early collapse on tool_hang, while the final performance is unchanged; on kitchen-partial, removing it improves early learning and leads to a similar endpoint. I therefore find the central argument incomplete: the paper identifies a failure mode and proposes a plausible remedy, but does not sufficiently explain when this remedy is preferable or whether the advantage comes from its specific design rather than more conservative updates in general. The stronger results of the complete SFG configuration do not by themselves resolve this attribution issue.

Questions for authors

(a) Under what conditions should the functional-step constraint help, and when should it be expected to slow learning? Can the authors connect the contrasting results on tool_hang, kitchen-partial, and transport to properties of the task or the pretrained policy, and provide evidence that this explanation extends beyond the two ablation tasks?

(b) What advantage does episode-level projection provide over a penalty on action changes relative to the previous policy or another adaptive update constraint? A matched comparison examining early performance loss, learning efficiency, and final performance would materially affect my assessment.

(c) Should the main contribution be understood primarily as preserving pretrained competence during the initial phase, or as improving the broader effectiveness of flow-policy fine-tuning? If the latter, what evidence isolates that benefit from the other components of the SFG configuration?

3 comments

We thank Reviewer VoEh for the detailed assessment and for valuing our report of the limits. Changes are in blue in the revised PDF.

W(a). The experimental coverage is too limited for me to assess how broadly the method is useful and how large its benefits are likely to be. The main comparison averages over five task settings, three of which are variants of Franka-Kitchen, while the component ablations cover only two tasks. The additional transport and point-cloud experiments are useful, but they do not establish consistent gains across a sufficiently diverse set of conditions. In particular, the one-step method fails on transport, and the multi-step results involve different pretrained policies and additional implementation considerations. Broader evaluation on further Robomimic tasks and suitable OGBench tasks would help determine whether the findings extend beyond the current settings. This matters because the reported benefit appears to depend on the competence of the initial policy and the task: preserving an already useful policy and learning from a weak initialization may require different update behavior.

Q(a). Under what conditions should the functional-step constraint help, and when should it be expected to slow learning? Can the authors connect the contrasting results on tool_hang, kitchen-partial, and transport to properties of the task or the pretrained policy, and provide evidence that this explanation extends beyond the two ablation tasks?

 

Coverage. The revision extends every part of the evaluation:

ExperimentSubmissionRevision
Main results5 tasks7 tasks: + Robomimic can and lift (Figure 3, Table 1)
Ablations2 tasks3 tasks: + square (Table 2)
Point-cloud observations1 task4 tasks: + Adroit door, pen, relocate (Figure 6)
Multi-step flow policiesAppendix (Table 5), transportMain results on all seven tasks, from the same pretrained policy as OGPO+CA

Q(a). The bound costs little when the pretrained policy is weak and protects it when it is competent but fragile (Table 2, now on three tasks; first 0.3M with and without the bound):

  • Weak start. On square (θ0\theta_0 = 0.19) and kitchen-partial (0.32) the update without the bound does not collapse, and the bound does not slow learning (0.73 against 0.69, 0.70 against 0.65; it ends at 0.98 and 1.00).
  • Competent but fragile start. On tool_hang (0.50) and on transport with a three-step policy (0.54), the update without the bound is at zero for the first 0.3M or throughout; with it, tool_hang keeps 0.70 early and transport ends at 0.81.
  • Where it slows learning. Where larger steps are tolerated, a looser bound is faster: δ\delta = 0.05 is the fastest early on square and kitchen-partial but falls below θ0\theta_0 on tool_hang (0.37 against 0.50); δ\delta = 0.01 is safe on all three.

W(b). The paper motivates controlling policy changes, but does not convincingly establish why the proposed functional-step projection is the appropriate way to do so, or why it should be preferred over other constraints. The learning-rate ablations are helpful, but they do not resolve this question. Comparisons with an action-distance penalty relative to the previous policy, another trust-region formulation, or adaptive step-size control would be more informative about the specific contribution. These comparisons should use the same pretrained actor, critic, replay scheme, and moving reference policy. As it stands, the strongest controlled result is that the constraint avoids an early collapse on tool_hang, while the final performance is unchanged; on kitchen-partial, removing it improves early learning and leads to a similar endpoint. I therefore find the central argument incomplete: the paper identifies a failure mode and proposes a plausible remedy, but does not sufficiently explain when this remedy is preferable or whether the advantage comes from its specific design rather than more conservative updates in general. The stronger results of the complete SFG configuration do not by themselves resolve this attribution issue.

Q(b). What advantage does episode-level projection provide over a penalty on action changes relative to the previous policy or another adaptive update constraint? A matched comparison examining early performance loss, learning efficiency, and final performance would materially affect my assessment.

 

The revision adds this matched comparison (Appendix I.1): a penalty on the action change relative to the previous policy, and a parameter-space trust region, each with the same pretrained actor, critic, replay, learning rate, and moving reference (the actor at the start of the episode) as SFG. Their strengths are calibrated so that the committed step at the start of training matches δ\delta = 0.01. Means over the three ablation tasks (Table 9):

ConstraintFirst 0.3M (mean / lowest)Steps to 0.8 (M)AUC 0–0.5MFinal
SFG projection0.71 / 0.330.240.750.96
Penalty on action change0.68 / 0.310.350.750.95
Parameter-space trust region0.61 / 0.190.380.670.96
No constraint0.45 / 0.010.470.530.96
  • What the projection adds is precise control of the step. It holds the committed functional step at δ\delta throughout training (0.0100 to 0.0102); the penalty's step drifts (from 0.0086 to 0.0043 on tool_hang), and the trust region, which bounds parameters rather than actions, needs a different ϵ\epsilon on each task and still drifts up to 2.9 δ\delta. Adaptive control of δ\delta itself is left to future work.
  • The penalty is a viable alternative. It is close to SFG, faster on kitchen-partial and square and slower on tool_hang; on average the projection is slightly higher early and faster, with the same AUC and final score.
  • Not just more conservative updates. The smaller learning rates of Table 2, without the bound, stay below SFG early on tool_hang and kitchen-partial (at best 0.56 and 0.53 against 0.70) and do not end higher on any task.

Q(c). Should the main contribution be understood primarily as preserving pretrained competence during the initial phase, or as improving the broader effectiveness of flow-policy fine-tuning? If the latter, what evidence isolates that benefit from the other components of the SFG configuration?

 

Both, from different components. The bound's role is the first: it preserves the pretrained competence early and leaves the final score unchanged (Table 2; our answer to (a)). The broader effectiveness comes from the recipe as a whole, a direct deterministic QQ-gradient on a one-step generator with the bound, and the revised Table 1 now isolates it from the starting policy and the policy architecture. Seven-task means:

MethodInitial scoreCompletion scoreSteps to 0.8 (M)
OGPO+CA0.430.910.56
SFG (10-step flow)0.430.920.68
SFG (1-step flow)0.410.940.47
SFG (MeanFlow)0.430.980.21
  • Not the starting policy. All four start at 0.41 to 0.43.
  • The update rule. SFG (1-step flow) fine-tunes a one-step policy distilled from the same ten-step flow policy that OGPO+CA starts from, and still ends higher and reaches 0.8 sooner (0.94 and 0.47M against 0.91 and 0.56M).
  • The one-step generator. Applied directly to the ten-step flow policy, the same update ends at the level of OGPO+CA but is slower (0.68M), since the critic's gradient then passes through the sampler.
Reviewer_1JhiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper fine-tunes pretrained flow policies online with the plain deterministic policy gradient, without an anchor to the data. A study with a one-step flow policy shows that the first few updates erase the pretrained skill. Their method, Simple Flow Gradient (SFG), lets a proposal copy of the actor collect data and train freely during each episode. It then keeps only as much of that change as fits a small bound on how far the generated actions move. On five Robomimic and Franka-Kitchen tasks, the authors report that SFG beats every baseline on average and learns more than twice as fast.

Critical strengths
  1. The diagnostic study makes the early fine-tuning dip measurable, and it shows that parameter distance is a poor measure of how much an update changes the actions.

  2. The recipe is simple, with one extra hyperparameter, the same settings on every task and no actor anchor.

Critical weaknesses
  1. The main comparison doesn't isolate the update rule. Each method starts from its own behavior cloning policy, and in fig. 3b these starts differ a lot. That matters because the speed metric, steps to a fixed score, is very sensitive to the start. The baselines also run at published defaults, while the bound δ\delta was tuned on two of the five evaluation tasks. If OGPO+CA starts from the same OGPO checkpoints as the paper's ten-step SFG, table 5's column with the gradient through only the last sampler step is a rough matched comparison. By my calculation, the five-task means are then about the same for SFG and OGPO+CA. Much of the main-table gap could therefore come from SFG's stronger one-step pretrained policy. I recommend to compare SFG, the same update with a tuned anchor, and a one-step actor distilled from the flow policy, all from one pretrained policy and critic. Confidence intervals would also help, since the paper's own seed check shows a wide spread.

  2. The bound does less than the framing suggests. In table 2, removing it leaves the final scores essentially unchanged, so it mainly helps early. The bound also constrains only the committed actor, which is the one evaluated. The proposal that collects the data is unbounded within an episode. Early on tool_hang, it typically ends an episode far more than δ\delta away from the committed actor. So the claim that SFG's success "never drops below its starting point" says nothing about behavior on the robot during training.

Questions for authors
  1. Does OGPO+CA start from the same flow-matching checkpoints as the multi-step SFG, and with how many Euler steps? If so, I recommend to discuss the last-step column of table 5 against OGPO+CA's per-task results.

  2. Could you run the matched-start comparison from my first weakness on tool_hang and one kitchen task?

  3. What exactly was run for each baseline, and was any of them tuned on these tasks? How do your numbers compare with published results, where those exist? Did QC-FQL get its offline reinforcement learning phase?

  4. How does the success of the proposal actor's training rollouts change over time, especially early on tool_hang?

3 comments

We thank Reviewer 1Jhi for the specific review and for valuing the diagnostic study. Changes are in blue in the revised PDF.

W1. The main comparison doesn't isolate the update rule. Each method starts from its own behavior cloning policy, and in fig. 3b these starts differ a lot. That matters because the speed metric, steps to a fixed score, is very sensitive to the start. The baselines also run at published defaults, while the bound δ\delta was tuned on two of the five evaluation tasks. If OGPO+CA starts from the same OGPO checkpoints as the paper's ten-step SFG, table 5's column with the gradient through only the last sampler step is a rough matched comparison. By my calculation, the five-task means are then about the same for SFG and OGPO+CA. Much of the main-table gap could therefore come from SFG's stronger one-step pretrained policy. I recommend to compare SFG, the same update with a tuned anchor, and a one-step actor distilled from the flow policy, all from one pretrained policy and critic. Confidence intervals would also help, since the paper's own seed check shows a wide spread.

Q1. Does OGPO+CA start from the same flow-matching checkpoints as the multi-step SFG, and with how many Euler steps? If so, I recommend to discuss the last-step column of table 5 against OGPO+CA's per-task results.

Q2. Could you run the matched-start comparison from my first weakness on tool_hang and one kitchen task?

 

The revision adds the matched comparisons to the main results (Figure 3, Table 1), on all seven tasks rather than two. SFG (10-step flow) starts from the same flow-matching checkpoints as OGPO+CA, with ten Euler steps and the gradient through the last step (Q1); SFG (1-step flow) is the one-step actor distilled from that flow policy (Q2). Table 1 also gains an Initial score column, and Robomimic can and lift join the five tasks of the submission. Seven-task means:

MethodInitial scoreCompletion scoreSteps to 0.8 (M)
OGPO+CA0.430.910.56
SFG (10-step flow)0.430.920.68
SFG (1-step flow)0.410.940.47
SFG (MeanFlow)0.430.980.21
  • Starts and the speed metric. SFG's one-step pretrained policy is not stronger: all four rows start at 0.41 to 0.43, the same level as OGPO+CA, the fastest baseline. Steps to 0.8 therefore compare like for like against it (0.21M for SFG against 0.56M).
  • Ten-step SFG against OGPO+CA (Q1). Your calculation holds: from the same checkpoints, SFG ends at the level of OGPO+CA (0.92 against 0.91) and reaches 0.8 later (0.68M against 0.56M). This comparison is now in the main text rather than left to Table 5.
  • Distilled one-step actor (Q2). From the same source policy, SFG ends higher than OGPO+CA and reaches 0.8 sooner (0.94 and 0.47M against 0.91 and 0.56M).

Tuning. δ\delta = 0.01 is fixed on every task and experiment, including the multi-step and point-cloud runs; the δ\delta rows of Table 2 are ablations. On square, added to Table 2, all three values end at 0.98, and 0.01 is not the fastest early (first 0.3M: 0.71 / 0.73 / 0.79 for δ\delta = 0.005 / 0.01 / 0.05). The baseline settings are under Q3.

W2. The bound does less than the framing suggests. In table 2, removing it leaves the final scores essentially unchanged, so it mainly helps early. The bound also constrains only the committed actor, which is the one evaluated. The proposal that collects the data is unbounded within an episode. Early on tool_hang, it typically ends an episode far more than δ\delta away from the committed actor. So the claim that SFG's success "never drops below its starting point" says nothing about behavior on the robot during training.

Q4. How does the success of the proposal actor's training rollouts change over time, especially early on tool_hang?

 

We agree on both points.

  • The bound mainly helps early. In Table 2 it leaves the final scores unchanged (0.91 / 1.00 / 0.98 with it, 0.90 / 0.99 / 0.99 without, on tool_hang / kitchen-partial / square) and acts on the first 0.3M steps, most on tool_hang (0.70 against 0.00). Its role is to keep the pretrained competence while the critic and the policy improve, not to raise the final level.
  • The proposal is unbounded within an episode. "Never drops below its starting point" refers to the smoothed evaluations of the committed actor, not to the training rollouts; Sections 5.2 and 5.3 now say so. The success of every training episode is logged, so Q4 can be read off the tool_hang run of Table 2:
tool_hangProposal (training episodes)Committed (evaluations)
First 40k0.160.38
First 0.3M0.630.70
Last 0.2M0.900.91
  • Your concern holds at the start. Over the first 40k steps the proposal ends its episodes far from the committed actor and succeeds in 16% of them, against 0.50 for θ0\theta_0.
  • The phase is short. From 40 to 60k steps on, the proposal succeeds in 0.5 to 0.7 of its episodes, close to the committed actor. Without the bound, the actor that collects the data is the evaluated one, and it is at zero over the first 0.3M (Table 2).

Q3. What exactly was run for each baseline, and was any of them tuned on these tasks? How do your numbers compare with published results, where those exist? Did QC-FQL get its offline reinforcement learning phase?

 

  • What was run, and tuning. OGPO+CA, QC-FQL, DSRL, and EXPO ran in the OGPO authors' public codebase with its per-task configurations, the setting of the OGPO paper. DFP and MVP keep the offline data in the replay buffer, as in their papers. None of the baselines was tuned per task, and every method gets the same 1.4M online steps from its own behavior-cloning policy.
  • QC-FQL's offline phase. Yes. Its offline stage on the demonstration data, as configured in the OGPO codebase, runs for a fixed 200k steps before online RL and has converged by then on every task.
  • Published results. We compared every baseline with published results on the tasks they share. OGPO+CA, QC-FQL, DSRL, and EXPO end close to the values in the OGPO paper (means within 0.06 over the five shared tasks, read from its Figures 9 and 10). DFP and MVP were published on Robomimic square, where our runs do even better (0.97 and 0.98, against 0.93 and 0.79 to 0.93).
Reviewer_zzNFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an off-policy RL algorithm for fine-tuning pre-trained flow-matching policies using direct Q-gradients without BC regularization. The authors first analyse how direct Q-gradient updates can be unstable, and then propose SFG, where the key idea is to prevent large policy updates by introducing a functional step projection to prevent large policy updates. Experiments on Robomimic and Franka-Kitchen show that SFG outperforms several generative policy fine-tuning baselines in both sample efficiency and final task performance.

Critical strengths

The paper is overall well-written, and the proposed method is simple yet interesting. I particularly appreciate the analyses in Section 3, which provide a lot of insight.

Critical weaknesses

W1. The main concern I have is that the proposed method focuses on one-step MeanFlow policies, which avoid the BPTT difficulties associated with direct Q-gradient fine-tuning of multi-step flow policies. While this simplifies the problem, it remains unclear whether the proposed method can effectively address the instability of direct Q-gradient updates in more general multi-step settings. Moreover, the results in Figure 5 suggest that multi-step flow policies generally perform worse than the one-step baseline. This somewhat limits the general applicability of the proposed method.

W2. Related to the point above, the baseline comparisons are difficult to interpret, as the methods use different pretrained policy architectures. In particular, the main results of SFG are obtained using a one-step MeanFlow policy, while several baselines use multi-step policies. Therefore, it is unclear whether the performance gains come from the one-step policy architecture or the proposed RL method. For example, would it be possible to evaluate OGPO (or other baselines) using the same pretrained one-step MeanFlow policy to better isolate the contribution of the proposed method?

W3. Most of the evaluation is limited to low-dimensional, state-based simulation tasks, where I suspect that a one-step policy may already be sufficient. I am curious how SFG would perform on more challenging pixel-based manipulation tasks.

Questions for authors

Q1. I am curious whether the initial performance collapse in Figure 2 could be attributed to insufficient critic training. Would it be possible to vary the number of critic warm-up steps to investigate whether a better-trained critic could prevent the initial collapse, even without the proposed functional step bound?

2 comments

We thank Reviewer zzNF for the kind words on the method and Section 3. Changes are in blue in the revised PDF.

W1. The main concern I have is that the proposed method focuses on one-step MeanFlow policies, which avoid the BPTT difficulties associated with direct Q-gradient fine-tuning of multi-step flow policies. While this simplifies the problem, it remains unclear whether the proposed method can effectively address the instability of direct Q-gradient updates in more general multi-step settings. Moreover, the results in Figure 5 suggest that multi-step flow policies generally perform worse than the one-step baseline. This somewhat limits the general applicability of the proposed method.

 

The bound constrains only the generated action chunk, so it applies to a multi-step sampler unchanged (Appendix G). SFG on multi-step flow policies is now part of the main results: Figure 3 and Table 1 add SFG (10-step flow), which fine-tunes the ten-step flow-matching policy that OGPO+CA fine-tunes, from the same starting policy, and SFG (1-step flow), which fine-tunes a one-step policy distilled from it. Seven-task means (the revision adds Robomimic can and lift to the five tasks of the submission), with the new Initial score column of Table 1:

MethodInitial scoreCompletion scoreSteps to 0.8 (M)
OGPO+CA0.430.910.56
SFG (10-step flow)0.430.920.68
SFG (1-step flow)0.410.940.47
SFG (MeanFlow)0.430.980.21

SFG (10-step flow) is stable and ends at the level of OGPO+CA, but is slower, since the critic's gradient still passes through the sampler.

W2. Related to the point above, the baseline comparisons are difficult to interpret, as the methods use different pretrained policy architectures. In particular, the main results of SFG are obtained using a one-step MeanFlow policy, while several baselines use multi-step policies. Therefore, it is unclear whether the performance gains come from the one-step policy architecture or the proposed RL method. For example, would it be possible to evaluate OGPO (or other baselines) using the same pretrained one-step MeanFlow policy to better isolate the contribution of the proposed method?

 

We tried OGPO on our one-step MeanFlow policy, but its update is built around a multi-step denoising chain (Appendix H) and performed poorly there. We therefore ran the matched comparison the other way round, SFG from OGPO's pretrained policy:

  • SFG does not start from a stronger policy: 0.43, the same as OGPO+CA and below the 0.53 to 0.57 of DSRL, EXPO, and QC-FQL (Table 1).
  • From the same source policy as OGPO+CA, distilled to one step, SFG still ends higher and reaches 0.8 sooner (0.94 and 0.47M against 0.91 and 0.56M; table above).

W3. Most of the evaluation is limited to low-dimensional, state-based simulation tasks, where I suspect that a one-step policy may already be sufficient. I am curious how SFG would perform on more challenging pixel-based manipulation tasks.

 

Besides DexArt bucket, already in the submission, the revision adds three tasks with point-cloud observations from a single camera, Adroit door, pen, and relocate (Figure 6). The policy has the DP3 architecture, a 255M-parameter Diffusion Policy U-Net with a PointNet encoder, trained with the MeanFlow objective. SFG improves the pretrained policy on all four tasks, by 20 points on average:

TaskPretrainedSFGGain
DexArt bucket0.470.69+0.22
Adroit door0.800.96+0.16
Adroit pen0.450.75+0.30
Adroit relocate0.580.70+0.12
Mean0.580.78+0.20

Q1. I am curious whether the initial performance collapse in Figure 2 could be attributed to insufficient critic training. Would it be possible to vary the number of critic warm-up steps to investigate whether a better-trained critic could prevent the initial collapse, even without the proposed functional step bound?

 

No. On tool_hang without the bound, we trained the critic twenty times longer before the first actor update (200k instead of 10k warm-up updates). The collapse is unchanged (steps from the first actor update):

Critic warm-up0–100k100–300k300–500k500–600kFirst success
10k (Table 2)0.000.000.090.20320k
200k0.000.000.060.19420k
  • The critic had settled: its mean QQ levels off within the first 25–40k warm-up updates, yet success is already zero at the first evaluation, 20k actor steps later.
  • With the default 10k warm-up, the bound alone keeps the pretrained success (0.70 against 0.00 over the first 0.3M, Table 2): the collapse comes from the size of the step, as the frozen-critic probe of Figure 2b also shows.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.