- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies task-level degradation during online PPO fine-tuning of SmolVLA. A controller adjusts exploration noise using novelty and competence signals, without task labels. On seven LIBERO-10 tasks, it avoids collapse under the pooled definition in all three main seeds, while fixed and learned noise frequently lose tasks. No method improves on the initial behavior-cloning policy.
Critical strengths
-
Tracking individual tasks exposes failures that aggregate success can obscure. Reporting pooled, last-scan, sustained and ever-collapse measures gives a more informative picture of policy degradation.
-
The main comparison holds the policy and training setup fixed while changing exploration. The controller consistently preserves more competence than the evaluated fixed and learned-noise alternatives.
-
The paper carefully reports evaluation variability, parameter movement and negative results. Its discussion of numerical precision and the limitations of the fp32 pilot is useful and appropriately qualified.
Critical weaknesses
-
The experiments establish an effect of the complete controller, but not the contribution of its uncertainty signals. Novelty-only, competence-only and shuffled-gate controls are missing. Similar mean noise scales do not establish similar exploration budgets, since noise variance depends on squared scale.
-
Evidence for preserving skills during productive learning is still missing. No arm exceeds behavior cloning, the main experiments use in-place bfloat16 updates, and the fp32 pilot does not resolve the issue. The additional benchmarks also show no consistent benefit. The positive result therefore remains limited to one training regime.
-
The strongest fixed-noise alternative is not evaluated as thoroughly as the main arms. Fixed noise at 0.15 avoids pooled collapse in one seed but loses two tasks in another, with only two seeds reported. More repetitions could materially change the comparison with the controller.
Questions for authors
- Does the controller outperform a shuffled gate that preserves its noise distribution, or a time-dependent schedule with matched noise variance?
- Does the preservation advantage remain under a tuned training configuration that improves task performance above behavior cloning?
- How does fixed noise at 0.15 compare across the same seeds and evaluation budget as the controller?
3 comments
Thanks for running the extra controls. Answer2 answers my main question about the noise budget. It keeps the same noise scales at each step and changes which rollout gets which scale. The controller did better on two seeds, and the difference on the third was within the reported run-to-run variation. This suggests that it matters where the controller applies the noise. The comparison with fixed = 0.15 is also clearer now. With the same three seeds and evaluation budget, the controller had higher pooled success on all three, though the gap was small on one seed.
The FLOWER results answer part of my second question. Both methods improved over the starting policy on all three seeds, and neither lost a task. That shows the controller can preserve skills while the policy improves. I still don’t know whether it preserves more skills than fixed noise in that setting.
The question I still have is how much novelty and competence each contribute. The signal ablations were run in fp16, where even the full controller lost tasks. Do you have novelty-only and competence-only results in the original bf16 setting? That’s where the full controller avoided pooled collapse, so those runs would help me understand which signal accounts for the result.
Thank you for the follow-up. We do not have those runs in bfloat16. What we have there is the log of the controller's three seeds (Appendix D.2, Table D.1).
The controller applies a mean noise scale σ of 0.28 on the three lower-tare tasks and 0.18 on the three higher-tare tasks. The competence flag produces this difference. It is raised on 64% of the steps on the lower-tare tasks and on 19% on the higher-tare tasks. The novelty flag is raised on about 10% of all steps by design, most often on a different task of middle tare.
So competence decides which tasks get more noise, and novelty adds noise on the few steps with a high RND error.
Response to Reviewer MRBX
Thank you for the review.
Question 1. We added a control, S2, that we believe covers both alternatives you name. It keeps K's applied σ at every step (a time-dependent schedule with exactly matched variance) and shuffles it across the 14 rollout streams at every step (a shuffled gate with K's distribution); its mean σ and mean σ² equal K's. With three seeds (Table 2), S2 is 0.167 and 0.114 below K in pooled success on seeds 1337 and 3337 with one pooled collapsed task on each, while K has none on any seed. On seed 2337 the difference is within the run-to-run SD.
Question 2. We added that configuration on a second VLA policy, FLOWER. There, with seven LIBERO-90 tasks chosen to leave room above the starting policy (tares 0.69 to 0.75) and the main campaign's batch, learning rate and iteration count, both arms end above their starting point on all three seeds: over the last five scans the controller pools 0.867, 0.914 and 0.870 and fixed noise 0.864, 0.884 and 0.880 (Appendix G.4, second precision regime). The controller loses no task under any of the four definitions while success rises. Fixed noise loses none there either, so the preservation advantage is not tested in that setting; our reading, not tested, is that it does not put the policy under the pressure in which tasks are lost. In the paper's own setting no run exceeds its starting point.
Question 3. Fixed σ 0.15 now has K's three seeds and evaluation budget (Table 2). It loses pooled tasks on one of three seeds (2/7 on seed 2337). K is above it in pooled success on every seed, by 0.026, 0.101 and 0.042; the first is below the larger measured run-to-run SD (0.0375).
Signal ablations and precision (weaknesses 1 and 2). Novelty-only and competence-only ran with three seeds each in a second precision regime, fp16 in place (Appendix G). In that regime every arm loses tasks, the controller included (3/7, 1/7, 1/7), so the ablations are not informative there, and the result is stated for the paper's regime (Section 8).
Following your point on the squared scale, Table 2 reports mean σ² beside mean σ for every arm.
We hope this answers your questions and are glad to clarify anything further.