acceptodds
Under review as a conference paper at ICLR 2027

Uncertainty-Gated Exploration Noise Suppresses Task Collapse in Online RL Fine-Tuning of a Flow-Matching Vision-Language-Action Policy

Abstract

Online reinforcement learning (RL) fine-tuning of pretrained flow-matching vision-language-action (VLA) policies promises robots that keep learning after deployment, but continued updates frequently destroy competence on individual tasks while the aggregate still looks healthy. We study this failure mode, which we call task collapse, under a matched small-compute budget on LIBERO-10 with a 450M-parameter SmolVLA policy trained by PPO in the stochastic (SDE) sampling regime. Three exploration-noise policies differ in a single live variable: a fixed noise scale, a learned noise network in the style of ReinFlow, and an uncertainty-gated controller that redistributes exploration across task streams from task-agnostic novelty and competence signals, without task labels or episode boundaries. Under the pooled definition, fixed noise collapses tasks in two of three seeds and learned noise in every seed measured to iteration 200, while the controller collapses none in any of its three seeds. Measured parameter displacement confirms that the controller's action expert continues to change, while its mean applied noise is close to the fixed scale in the available logs. The matched comparison supports the controller's effect on task preservation; the separate contributions of its adaptation across states and over time are not disentangled. A lower fixed scale slows the decline but does not stop it. No arm improves on the behavior-cloning baseline in this budget. Two properties of that regime are measured beside this result rather than offered as its cause: following the reference recipe, training runs in bfloat16 with no fp32 master copy, under which 96.02% of the action expert's elements stay bit-identical across three consecutive iterations, and an fp32 master copy at the reference learning rate collapses both arms in a single-seed observation. We release measurement tools for per-task collapse under four definitions, run-to-run rescoring noise and instrument tares.

Primary area:
reinforcement learning
Keywords:
vision-language-action models, online reinforcement learning, flow matching, exploration noise, task collapse, continual learning, robot learning, numerical precision

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MRBXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies task-level degradation during online PPO fine-tuning of SmolVLA. A controller adjusts exploration noise using novelty and competence signals, without task labels. On seven LIBERO-10 tasks, it avoids collapse under the pooled definition in all three main seeds, while fixed and learned noise frequently lose tasks. No method improves on the initial behavior-cloning policy.

Critical strengths
  1. Tracking individual tasks exposes failures that aggregate success can obscure. Reporting pooled, last-scan, sustained and ever-collapse measures gives a more informative picture of policy degradation.

  2. The main comparison holds the policy and training setup fixed while changing exploration. The controller consistently preserves more competence than the evaluated fixed and learned-noise alternatives.

  3. The paper carefully reports evaluation variability, parameter movement and negative results. Its discussion of numerical precision and the limitations of the fp32 pilot is useful and appropriately qualified.

Critical weaknesses
  1. The experiments establish an effect of the complete controller, but not the contribution of its uncertainty signals. Novelty-only, competence-only and shuffled-gate controls are missing. Similar mean noise scales do not establish similar exploration budgets, since noise variance depends on squared scale.

  2. Evidence for preserving skills during productive learning is still missing. No arm exceeds behavior cloning, the main experiments use in-place bfloat16 updates, and the fp32 pilot does not resolve the issue. The additional benchmarks also show no consistent benefit. The positive result therefore remains limited to one training regime.

  3. The strongest fixed-noise alternative is not evaluated as thoroughly as the main arms. Fixed noise at 0.15 avoids pooled collapse in one seed but loses two tasks in another, with only two seeds reported. More repetitions could materially change the comparison with the controller.

Questions for authors
  1. Does the controller outperform a shuffled gate that preserves its noise distribution, or a time-dependent schedule with matched noise variance?
  2. Does the preservation advantage remain under a tuned training configuration that improves task performance above behavior cloning?
  3. How does fixed noise at 0.15 compare across the same seeds and evaluation budget as the controller?
3 comments
AuthorsOpenReview ↗

Response to Reviewer MRBX

Thank you for the review.

Question 1. We added a control, S2, that we believe covers both alternatives you name. It keeps K's applied σ at every step (a time-dependent schedule with exactly matched variance) and shuffles it across the 14 rollout streams at every step (a shuffled gate with K's distribution); its mean σ and mean σ² equal K's. With three seeds (Table 2), S2 is 0.167 and 0.114 below K in pooled success on seeds 1337 and 3337 with one pooled collapsed task on each, while K has none on any seed. On seed 2337 the difference is within the run-to-run SD.

Question 2. We added that configuration on a second VLA policy, FLOWER. There, with seven LIBERO-90 tasks chosen to leave room above the starting policy (tares 0.69 to 0.75) and the main campaign's batch, learning rate and iteration count, both arms end above their starting point on all three seeds: over the last five scans the controller pools 0.867, 0.914 and 0.870 and fixed noise 0.864, 0.884 and 0.880 (Appendix G.4, second precision regime). The controller loses no task under any of the four definitions while success rises. Fixed noise loses none there either, so the preservation advantage is not tested in that setting; our reading, not tested, is that it does not put the policy under the pressure in which tasks are lost. In the paper's own setting no run exceeds its starting point.

Question 3. Fixed σ 0.15 now has K's three seeds and evaluation budget (Table 2). It loses pooled tasks on one of three seeds (2/7 on seed 2337). K is above it in pooled success on every seed, by 0.026, 0.101 and 0.042; the first is below the larger measured run-to-run SD (0.0375).

Signal ablations and precision (weaknesses 1 and 2). Novelty-only and competence-only ran with three seeds each in a second precision regime, fp16 in place (Appendix G). In that regime every arm loses tasks, the controller included (3/7, 1/7, 1/7), so the ablations are not informative there, and the result is stated for the paper's regime (Section 8).

Following your point on the squared scale, Table 2 reports mean σ² beside mean σ for every arm.

We hope this answers your questions and are glad to clarify anything further.

Thanks for running the extra controls. Answer2 answers my main question about the noise budget. It keeps the same noise scales at each step and changes which rollout gets which scale. The controller did better on two seeds, and the difference on the third was within the reported run-to-run variation. This suggests that it matters where the controller applies the noise. The comparison with fixed σ\sigma = 0.15 is also clearer now. With the same three seeds and evaluation budget, the controller had higher pooled success on all three, though the gap was small on one seed.

The FLOWER results answer part of my second question. Both methods improved over the starting policy on all three seeds, and neither lost a task. That shows the controller can preserve skills while the policy improves. I still don’t know whether it preserves more skills than fixed noise in that setting.

The question I still have is how much novelty and competence each contribute. The signal ablations were run in fp16, where even the full controller lost tasks. Do you have novelty-only and competence-only results in the original bf16 setting? That’s where the full controller avoided pooled collapse, so those runs would help me understand which signal accounts for the result.

Thank you for the follow-up. We do not have those runs in bfloat16. What we have there is the log of the controller's three seeds (Appendix D.2, Table D.1).

The controller applies a mean noise scale σ of 0.28 on the three lower-tare tasks and 0.18 on the three higher-tare tasks. The competence flag produces this difference. It is raised on 64% of the steps on the lower-tare tasks and on 19% on the higher-tare tasks. The novelty flag is raised on about 10% of all steps by design, most often on a different task of middle tare.

So competence decides which tasks get more noise, and novelty adds noise on the few steps with a high RND error.

Reviewer_ZeggOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies task degradation during online RL fine-tuning of SmolVLA. It adjusts exploration noise using novelty and critic signals, reducing task collapse on seven LIBERO tasks under the evaluated training budget.

Critical strengths

s1. The controller reduces degradation and gives more consistent results across three seeds than the evaluated fixed and learned noise baselines.

s2. The paper clearly reports negative results and limitations.

Critical weaknesses

w1. It is not yet clear whether the uncertainty signals drive the gains. A simpler way of varying noise might work just as well.

w2. All methods perform worse than the initial BC policy. I would like to see whether the controller can preserve skills while also improving overall performance.

w3. The positive results come from one VLA and seven tasks. Results elsewhere are mixed, and there is no real-robot evaluation.

Questions for authors

q1. Could you shuffle noise assignments across environments while keeping the same distribution? This would help show whether matching noise to the current state matters.

q2. Could you test the controller in a setting where RL improves overall success, and check whether it still prevents individual tasks from degrading?

q3. Could you add a small real-robot evaluation to see whether the same protection holds in practice?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Zegg

Thank you for the review. We answer in your numbering.

q1. We added the shuffle you describe as a new control, S2. It replays the σ values the controller K applied to its 14 rollout streams and shuffles them across the streams at every step, so the distribution, its timing and both moments stay K's. With three seeds (Table 2), S2 is 0.167 and 0.114 below K in pooled success on seeds 1337 and 3337 and loses one pooled task on each, while K loses none on any seed. On seed 2337 the difference is within the run-to-run SD.

q2. We added that setting on a second VLA policy, FLOWER. There, with seven LIBERO-90 tasks chosen to leave room above the starting policy (tares 0.69 to 0.75) and the main campaign's batch, learning rate and iteration count, both arms end above their starting point on all three seeds: over the last five scans the controller pools 0.867, 0.914 and 0.870 and fixed noise 0.864, 0.884 and 0.880 (Appendix G.4, second precision regime). The controller loses no task under any of the four definitions while success rises. Fixed noise loses none there either, so the two arms are not separated in that setting; our reading, not tested, is that it does not put the policy under the pressure in which tasks are lost. In the paper's own setting no run exceeds its starting point, including from a second base policy trained on 10 demonstrations per task (Appendix G.2), where K ends above fixed σ 0.25 on all three seeds.

q3. We were not able to add a real-robot evaluation in this revision. The added breadth is the second VLA policy above, on the paper's pool (one seed per arm, starting near the ceiling) and on the LIBERO-90 pool (three seeds per arm).

Changes in the revised PDF are in blue (Section 6, Table 2, Appendices F and G).

We hope this answers your questions and are glad to clarify anything further.

Reviewer_YZ6vOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies loss of individual task competence during PPO fine-tuning of SmolVLA on seven jointly trained LIBERO tasks. It compares fixed noise, a ReinFlow-style learned noise network, and a controller driven by novelty and competence statistics. Three seeds per arm are followed for 200 iterations. The paper also measures evaluation variability and parameter movement under the bfloat16 training recipe.

Critical strengths
  1. The per-task analysis exposes failures that aggregate success can hide. Under the primary pooled definition, collapse occurs in none of the three controller seeds, two of three fixed-noise seeds, and all three learned-noise seeds (Table 1). The alternative collapse definitions and common evaluation horizon make the conclusion less dependent on a single final checkpoint.

  2. The released records make the main measurements easy to check. The supplied episode logs reproduce the main pooled success counts, and the paper distinguishes evaluation noise from training-seed variation. It also states that no arm improves over behavior cloning and treats the precision pilot as an observation rather than a controlled explanation of collapse.

Critical weaknesses
  1. The results support preservation by the evaluated controller as a whole, but do not isolate the value of its novelty and competence signals. A similar mean noise scale does not match the distribution or timing of injected noise. The lower fixed-scale runs in Section 6 are useful, but also do not separate informative allocation from a different noise schedule. Appendix D.1 explicitly acknowledges that state allocation and adaptation over time are not disentangled. The gains could therefore reflect a different allocation of noise without requiring these particular signals.
Questions for authors
  1. Can the authors compare the controller with a control that preserves the distribution of applied noise scales at each environment step but randomly assigns those scales across rollout streams? This would retain the timing and noise budget, including its second moment, while testing the value of assigning noise according to the measured state signals.
1 comment
AuthorsOpenReview ↗

Response to Reviewer YZ6v

Thank you for the review. You asked for a control that keeps the distribution of applied noise scales at each environment step and assigns those scales randomly across rollout streams. Following your suggestion, we added this control (S2) and ran it in the paper's setup with three seeds and 200 iterations each.

  • What S2 does. It replays, step by step, the 14 values the controller K applied in the run of the same seed, and shuffles them across the 14 streams with a fresh permutation at every step. Timing, mean σ and mean σ² equal K's (0.2368, 0.2358, 0.2380 and 0.07105, 0.07046, 0.07152). No stream keeps a noise level of its own under S2: the spread of the 14 per-stream mean σ falls from 0.163, 0.130 and 0.146 under K to 0.002, 0.003 and 0.002 (Appendix F.1, Table 7).
  • Result (Table 2). On seeds 1337 and 3337, S2 is 0.167 and 0.114 below K of the same seed in pooled success and loses one pooled task on each; K loses none on any seed. On seed 2337 the difference (0.003) is within the run-to-run SD (0.045).

In the paper: Section 6, Table 2 and Appendix F. Appendix D.1 now says that allocation across streams is tested. Changes are in blue.

We hope this answers your questions and are glad to clarify anything further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

We thank the reviewers for their careful reading. Each review asked whether the controller's effect comes from where the noise goes or only from how much noise there is. The revision adds a control for that question. S2 shuffles the controller's applied σ across its 14 rollout streams at every step and keeps the values, their timing and both moments. With three seeds (Section 6, Table 2, Appendix F), S2 is 0.167 and 0.114 below the controller in pooled success on two seeds and loses a task on each, while the controller loses none; on the third seed the difference is within the run-to-run SD.

Also added: a third seed of fixed σ 0.15 (Table 2), a second base policy trained on fewer demonstrations (Appendix G.2), signal ablations in a second precision regime (Appendix G.1), and a second VLA policy, FLOWER, including a task pool on which both arms improve on the starting policy and neither loses a task (Appendix G.4). Section 6 now reports the controller's mean applied noise over all 200 iterations of its three seeds (0.2368, 0.2358, 0.2380). Changes in the revised PDF are in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.