- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
AnchorQ proposes a new online RL method for VLA finetuning.
AnchorQ uses flow reversal to map actions back into the noise space (namely noise anchors), where perturbations can be applied to sample new actions for exploration. A Q function is learned to score and select anchors. The paper provides theoritical studies and conduct empirical studies on libero, robotwin, and 3 real robot experiments.
Critical strengths
- The paper is well written and technically sound. The real robot RL results are promising.
- The core idea using flow reversal to construct a structured action space for more efficient exploration is intuitive and well motivated.
Critical weaknesses
- Missing ablations. AnchorQ is compared to only one fixed beta which does not prove the advantage of adaptive beta based on uncertainty estimation. Also it seems the weighted averaging has not been ablated. A possible way is to use particles with higest Q to ablate the effect of weighted averaging.
- Fig 8 introduces "no demo" but didn't explain it adequately in texts or captions. Could the author provide more clarifications about "no demo" and prefilling setup for the methods involved
Questions for authors
See weaknesses
2 comments
Response
Thank the authors for the adding new experiments and clarifications which are informative and have addressed all my concerns. My recommendation remains acceptance.
We thank the reviewer for recognizing our contributions and for suggesting these ablations.
We have conducted the suggested ablation using the same trained critic ensemble:
Weighted averaging improves success on all three suites. We have included this comparison in Appendix D.1, Table 5.
We conducted sweeps over β ∈ {0.1, 1, 3, 10} and σ ∈ {0.3, 0.5, 1.0}. For each suite, we selected the best-performing fixed combination (β = 0.1 and σ = 0.3) for the reported ablation. We have stated the sweep ranges explicitly in Appendix C.2 and clarified this selection in the ablation table caption. These results show the advantage of adapting β and σ over the best fixed combination from these sweeps.
Only the Hard-10 suite uses additional demonstrations. We add one successful trajectory per task in replay in the demo-assisted comparisons, following FRS. For DSRL + FRS, we use the same implementation described by Tang et al.: an auxiliary behavior-cloning (BC) actor loss is computed on demonstration transitions sampled from the buffer, using flow-reversed demonstration noises as targets. Please refer to the FRS paper for implementation details. AnchorQ uses the demonstration for replay prefill and candidate-Q supervision. “No demo” removes both replay prefill and the demonstration loss. We have clarified this in Appendix C.2 and the caption of Fig. 8.
We hope these address the reviewer's concerns and welcome any further conversation.
References