- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies reinforcement-learning fine-tuning of vision-language-action (VLA) models under non-episodic scene resets. Standard episodic RL resets the scene every episode, which is costly in the real world, while prior non-episodic RL mostly trains small policies from scratch. The authors introduce CRONOS, a ManiSkill3 benchmark with 4 scenes and 16 “put the [object] on the [receptacle]” tasks that share scenes, so manipulation errors compound across tasks. They formalize NSR-MTRL and fine-tune OpenVLA-7B and SpatialVLA-4B with PPO. Three findings follow: (1) naive fine-tuning collapses as the reset interval grows from 80 to 320 and 2560 steps (OpenVLA success 65%→37%→0%); (2) end-effector reset (EER), a curriculum of frequent early resets, and a heuristic that resets unrecoverable scenes using privileged simulator state (HSR, 14%→64%) recover performance, whereas learned resets (LSR, PER) do not; (3) at the same interaction budget, non-episodic fine-tuning outperforms episodic fine-tuning on out-of-domain layouts and sequential tasks.
Critical strengths
- The motivation is right. Scene resets are a main obstacle to scaling RL fine-tuning of VLAs on real robots, and existing non-episodic methods focus on small policies trained from scratch.
- The failure diagnosis is clear. End-effector drift away from the pretraining pose, and object positions that disperse and eventually leave the workspace as the reset interval grows (Figures 3 and 6), turn “it fails out of the box” into distribution shifts one can intervene on.
- The generalization result is valuable. At the same interaction budget, non-episodic fine-tuning is stronger on overlapping and cluttered layouts and on later tasks in a sequence, with nearly 2×–3× the success of episodic fine-tuning by the fourth task (Table 1, Figure 10).
- The limitations are stated directly: HSR uses privileged state, scene drift remains under long intervals, and contact-rich tasks are left for future work. Algorithm 1 and hyperparameter Table 2 are provided.
Critical weaknesses
- As a benchmark, coverage is too narrow. All 16 tasks are the same pick-and-place template, with 2 objects and 2 receptacles per scene, and no contact-rich, deformable, tool-use, or articulated objects. Only OpenVLA-7B and SpatialVLA-4B are tested, and only with PPO. There is no newer VLA and no small from-scratch baseline to support the claim that VLA fine-tuning behaves differently.
- The strongest improvement does not transfer. HSR’s 14%→64% uses simulator ground truth; LSR and PER fail. The paper mainly shows that an oracle reset works and learned resets do not, which does not solve the real-world reset problem it poses.
- Comparisons are not normalized by reset budget, and Section 5 lacks a state-diversity-matched control, so the cause of the generalization gain is unclear.
- Statistical reporting is insufficient, reward and success are undefined, and the formulation does not match the algorithm.
Questions for authors
- How many scene resets do PR, HSR, HSR-only, LSR, and PER actually trigger during training? Please add a success-versus-reset-count curve at a matched reset budget.
- Is the reward sparse success or shaped and dense? What geometric threshold defines success?
- How many random seeds are behind Table 1 and Figure 10? Please report mean and standard deviation.
- If HSR uses only visual observations and no privileged state, how much does success drop?
4 comments
Question 1
How many scene resets do PR, HSR, HSR-only, LSR, and PER actually trigger during training? Please add a success-versus-reset-count curve at a matched reset budget.
Table A reports the success rate at matched reset counts, and Table B reports the number of scene resets triggered within the same 1.31M interaction steps. Resetting unrecoverable scenes does not add overhead to the reset budget, as HSR achieves a higher success rate for the same number of resets used.
Table A. Success at matched reset counts (in-domain success rate %).
| # Resets | PR | HSR | HSR-only | LSR | PER | CL |
|---|---|---|---|---|---|---|
| 250 | 11.2 ± 4.7 | 14.9 ± 3.3 | 11.2 ± 8.6 | 9.5 ± 8.6 | 12.1 ± 8.6 | 11.8 ± 2.3 |
| 500 | 13.6 ± 6.2 | 16.4 ± 2.8 | 9.9 ± 8.2 | 12.7 ± 12.3 | 1.1 ± 0.0 | 10.4 ± 1.0 |
| 750 | 8.2 ± 7.1 | 25.5 ± 2.7 | 20.8 ± 9.4 | 13.7 ± 13.7 | 2.1 ± 1.7 | 15.3 ± 0.7 |
| 1000 | 10.2 ± 5.7 | 47.0 ± 11.3 | 34.0 ± 23.4 | 1.2 ± 1.2 | 0.7 ± 0.2 | 20.2 ± 1.5 |
Table B. Reset cost at matched interaction steps (1.31M steps).
| Method | # Scene resets | Heuristic / Periodic | Heuristic trigger rate | Avg. steps between resets | Success rate (%) |
|---|---|---|---|---|---|
| PR | 512 ± 0 | 0 / 512 | – | 2,560 | 13.9 ± 7.2 |
| HSR | 1,270 ± 122 | 758 / 512 | 4.6% | 1,041 | 64.8 ± 2.0 |
| HSR-only | 1,044 ± 62 | 916 / 128 | 5.6% | 1,261 | 41.6 ± 31.4 |
| LSR | 512 ± 0 | 0 / 512 | – | 2,560 | 12.7 ± 12.3 |
| PER | 512 ± 0 | 0 / 512 | – | 2,560 | 1.0 ± 0.2 |
| CL | 2,304 ± 0 | 0 / 2,304 | – | 569 | 39.8 ± 12.5 |
Question 2
Is the reward sparse success or shaped and dense? What geometric threshold defines success?
Sparse rewards are given for grasping the object, maintaining the grasp, and successfully placing it on the target receptacle.
Success is defined by both geometric and contact thresholds. The object must be horizontally within half the receptacle's maximum bounding-box extent plus 1 cm, vertically within 5 cm of the receptacle surface, and exert a contact force greater than 0.03 N.
Question 3
How many random seeds are behind Table 1 and Figure 10? Please report mean and standard deviation.
Table 1 is averaged over 3 random seeds, and Figure 10 over 3 random seeds. We report mean ± standard deviation across seeds in Table D.
| Model | T | Overlap | Cluttered |
|---|---|---|---|
| OpenVLA | 80 | 0.554 ± 0.059 | 0.494 ± 0.050 |
| 320 | 0.593 ± 0.002 | 0.536 ± 0.039 | |
| SpatialVLA | 80 | 0.552 ± 0.097 | 0.512 ± 0.077 |
| 320 | 0.643 ± 0.135 | 0.555 ± 0.137 |
Seen task sequences :
| Model | T | Task 1 | Task 2 | Task 3 | Task 4 |
|---|---|---|---|---|---|
| OpenVLA | 80 | 0.572 ± 0.091 | 0.262 ± 0.088 | 0.129 ± 0.055 | 0.055 ± 0.039 |
| 320 | 0.650 ± 0.003 | 0.406 ± 0.017 | 0.221 ± 0.014 | 0.096 ± 0.008 | |
| SpatialVLA | 80 | 0.736 ± 0.052 | 0.180 ± 0.144 | 0.117 ± 0.149 | 0.037 ± 0.052 |
| 320 | 0.711 ± 0.105 | 0.332 ± 0.298 | 0.188 ± 0.215 | 0.119 ± 0.157 |
Unseen task sequences :
| Model | T | Task 1 | Task 2 | Task 3 | Task 4 |
|---|---|---|---|---|---|
| OpenVLA | 80 | 0.535 ± 0.073 | 0.208 ± 0.049 | 0.080 ± 0.014 | 0.030 ± 0.011 |
| 320 | 0.606 ± 0.032 | 0.288 ± 0.024 | 0.148 ± 0.023 | 0.066 ± 0.011 | |
| SpatialVLA | 80 | 0.572 ± 0.118 | 0.171 ± 0.142 | 0.047 ± 0.064 | 0.022 ± 0.031 |
| 320 | 0.626 ± 0.129 | 0.255 ± 0.206 | 0.118 ± 0.145 | 0.073 ± 0.096 |
Question 4
If HSR uses only visual observations and no privileged state, how much does success drop?
In our study, HSR is used to show that when to reset matters, not how to detect that a reset is needed. We therefore use privileged state as an oracle trigger, which isolates the effect of reset timing from errors in failure detection. Detecting unrecoverable states from visual observations is an orthogonal problem and we leave this to future work.
We thank the reviewer for the review. We have posted our response and updated the PDF. Please let us know if any concerns remain; we are happy to clarify.
We have updated the manuscript, with changes highlighted in blue in Table 1 and the appendix. These revisions also incorporate suggestions from the other reviewers.
Weakness 1
Our evaluation scope is consistent with recent RL-for-VLA studies accepted at top venues, which also centre on OpenVLA and pick-and-place tasks [1, 2].
Our paper studies reinforcement learning fine-tuning of pre-trained robot foundation models under non-episodic settings. A from-scratch baseline would therefore address a different problem that falls outside the scope of this work.
References [1] J. Liu, F. Gao, B. Wei, X. Chen, Q. Liao, Y. Wu, C. Yu, and Y. Wang. What Can RL Bring to VLA Generalization? An Empirical Study. In Advances in Neural Information Processing Systems (NeurIPS), 2025.
[2] L. Shi, S. Chen, F. Gao, Y. Chen, K. Chen, T. Zhang, H. Zang, W. Zhang, C. Yu, and Y. Wang. Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Models. In Conference on Robot Learning (CoRL), 2026.
Weakness 2
CRONOS is primarily an analysis study of the limitations of non-episodic RL fine-tuning for pre-trained VLAs, rather than a method that solves the real-world reset problem. LSR and PER fail because their additional reset tasks are as sample-inefficient to learn as the original tasks; improving sample efficiency with model-based RL [1] is a promising direction.
We compared the ground-truth detector with an off-the-shelf VLM (Qwen3-VL-4B, zero-shot, without fine-tuning) on 4,608 frames covering all four scenes, with objects placed across a grid of table positions. The VLM achieves an accuracy of 0.903 and an F1 of 0.918, missing only 98 of 2,585 unrecoverable scenes (recall 0.962). Of these 98 misses, 93 are objects still on the table near the workspace boundary, and only 5 are objects that have fallen off the table. These results suggest that HSR does not need to rely on a privileged detector.
[1] Z. Yang, T. M. Moerland, M. Preuss, A. Plaat, and E. S. Hu. Reset-free reinforcement learning with world models. Transactions on Machine Learning Research, 2025.
Weakness 3
Non-episodic training exposes the policy to diverse yet temporally correlated object configurations, which can facilitate learning. In contrast, episodic training independently resamples object configurations at the start of each episode. As shown in the table below, episodic training with object positions uniformly sampled across the workspace underperforms non-episodic training under the evaluation setting of Section 5.1. We report results averaged over three seeds per method, each trained for 655K interaction steps.
Weakness 4
Following Gupta et al. [1], we formulate the task distribution p(·) as a general sampling scheme that is not restricted to resampling at every time step. We instead maintain each task for 80 steps to match the evaluation episode length, since changing the task objective at every step prevents effective policy rollout and learning.
Reference [1] A. Gupta, J. Yu, T. Z. Zhao, V. Kumar, A. Rovinsky, K. Xu, T. Devlin, and S. Levine. Reset-free reinforcement learning via multi-task learning: Learning dexterous manipulation behaviors without human intervention. In IEEE International Conference on Robotics and Automation (ICRA), pp. 6664–6