- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies continual reinforcement learning for VLA models. It combines a frozen retention critic and a trainable adaptation critic with replay and policy regularization to learn new tasks while reducing forgetting. Experiments cover LIBERO, ManiSkill, and physical robots.
Critical strengths
- The evaluation is reasonably comprehensive. It includes online and offline learning, different models, and multiple robot platforms.
- Some settings show meaningful improvements. The method performs well under visual disturbances, and average success in the five-task physical experiment improves from 34% to 58%.
Critical weaknesses
- Combining a frozen historical critic and a trainable critic with replay and policy regularization follows a familiar retention strategy in continual learning. The paper does not clearly establish a substantial methodological advance.
- In Table 2, success improves only from 81.2% for sequential learning to 81.6%, while forgetting becomes worse. Although the five-task physical experiment improves average success, the earliest task still falls to 0%. It remains unclear when the method better preserves previous capabilities.
- The method combines value retention, replay, and policy regularization, with different implementations across backends. Without matched single-critic comparisons, it is difficult to determine whether the gains come from the dual-critic design.
Questions for authors
- How do you explain the worse forgetting and transfer metrics in Table 2? Under which conditions does the method clearly outperform standard sequential RL?
- How does performance change in each backend when the frozen retention critic is removed, with replay, policy regularization, and training budgets held fixed?
Response to Reviewer bDdd: Retention Across Settings and Single-Critic Comparison
Thank you for recognizing the breadth of evaluation and the gains under visual disturbances and in the five-task robot study. We address when retention helps and what happens without the frozen critic, including all additional benchmark comparisons.
Q1: Table 2 and conditions for improvement. On the original five-task Spatial protocol, CRL-VLA and SL achieve 81.6% and 81.2% final success, respectively. NBT is 4.2 versus 0.3, and FWT is 0.13 versus 3.9. Thus, similar final success coexists with worse retention and forward-transfer metrics in this setting. SL already exhibits little forgetting here.
After review, we added CRL-VLA evaluations on Long. The tables include every reference baseline in the expanded Table 2; baseline values are reproduced from the cited sources. Values are in %, with AVG measuring final success, NBT retention (lower is better), and FWT forward transfer.
LIBERO-Long
On Long, mean AVG exceeds SL by 4.2 points and ER by 5.2 points, with NBT close to SL (−2.3 versus −2.4).
Visual perturbations provide direct evidence for historical retention in the evaluated setting. In our additional three-task Spatial study, replay-only ER has FAR 0.611 and mean forgetting 0.22. Historical KL alone yields 0.889 and 0.05; frozen-teacher Q retention alone yields 0.868 and 0.04. Both FAR improvements over ER occur in all three seeds. Full achieves the highest mean FAR, 0.917±0.07.
The earliest task reaching 0%. This result is from the five-task offline robot experiment using WAV. T1 full-task success falls from 50% at checkpoint 4 to 0% at checkpoint 5 (Table 12), while grasp success at that final checkpoint remains 100% (Table 13). Failures therefore occur after grasping, during task completion.
Final T2–T5 success is 40%, 100%, 90%, and 60% for CRL-VLA, versus 0%, 50%, 50%, and 40% for the baseline. Overall final success increases from 34% to 58%. We will present this aggregate improvement alongside T1's trajectory. Each checkpoint–task pair has ten evaluation trials.
Q2: Removing the frozen critic. We implemented your suggested single-critic setting as C in OpenVLA-OFT/RIPT: one critic learns from historical and current data, retaining replay and historical KL but removing the frozen retention teacher. This also addresses AjcJ's component-analysis request and kEhZ's request to distinguish value retention from policy constraints.
Variants share the perturbed three-task Spatial sequence, training budget, and replay of the previous stage's latest 30 successful trajectories. Each seen task receives 16 evaluations after each stage. Results are mean±SD over three seeds.
Full's mean FAR is 25.0 points above C. C is also below A and D in all three seeds. These results support a stable historical reference in the evaluated configurations. We will report each variant's actual loss coefficients to make the comparison conditions explicit.
The new single-critic experiment covers RIPT. WAV uses a frozen value expert for historical supervision; Expo-FT combines frozen and trainable Q branches for candidate evaluation. We will distinguish these implementations and tie each quantitative conclusion to its evaluated backend.