acceptodds
Under review as a conference paper at ICLR 2027

CRONOS: Benchmarking Reset-free Reinforcement Learning for Robotic Manipulation

Abstract

Reinforcement learning (RL) is promising for adapting robot policies to unstructured real-world environments. However, standard RL pipelines rely on episodic training with frequent scene resets, which is impractical for real-world deployment due to the need for substantial human intervention. While prior work has studied non-episodic RL in relatively small policies trained from scratch, its behavior when fine-tuning modern large-scale pre-trained vision-language-action (VLA) models remains underexplored. In this work, we introduce CRONOS, a simulation benchmark for RL fine-tuning of VLAs under non-episodic scene resets. CRONOS leverages high-fidelity physics simulators and adopts shared-scene multi-task settings. Using CRONOS, we show that naively fine-tuning pre-trained policies fails in non-episodic settings due to shifts in object configurations and end-effector poses; however, these challenges can be mitigated through targeted reset allocation and by addressing biases in pre-trained models. Moreover, we find that non-episodic training enhances long-horizon manipulation and improves generalization to previously unseen object configurations and task sequences.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Reset-free Reinforcement Learning, Robotics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_r4P1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reinforcement-learning fine-tuning of vision-language-action (VLA) models under non-episodic scene resets. Standard episodic RL resets the scene every episode, which is costly in the real world, while prior non-episodic RL mostly trains small policies from scratch. The authors introduce CRONOS, a ManiSkill3 benchmark with 4 scenes and 16 “put the [object] on the [receptacle]” tasks that share scenes, so manipulation errors compound across tasks. They formalize NSR-MTRL and fine-tune OpenVLA-7B and SpatialVLA-4B with PPO. Three findings follow: (1) naive fine-tuning collapses as the reset interval grows from 80 to 320 and 2560 steps (OpenVLA success 65%→37%→0%); (2) end-effector reset (EER), a curriculum of frequent early resets, and a heuristic that resets unrecoverable scenes using privileged simulator state (HSR, 14%→64%) recover performance, whereas learned resets (LSR, PER) do not; (3) at the same interaction budget, non-episodic fine-tuning outperforms episodic fine-tuning on out-of-domain layouts and sequential tasks.

Critical strengths
  1. The motivation is right. Scene resets are a main obstacle to scaling RL fine-tuning of VLAs on real robots, and existing non-episodic methods focus on small policies trained from scratch.
  2. The failure diagnosis is clear. End-effector drift away from the pretraining pose, and object positions that disperse and eventually leave the workspace as the reset interval grows (Figures 3 and 6), turn “it fails out of the box” into distribution shifts one can intervene on.
  3. The generalization result is valuable. At the same interaction budget, non-episodic fine-tuning is stronger on overlapping and cluttered layouts and on later tasks in a sequence, with nearly 2×–3× the success of episodic fine-tuning by the fourth task (Table 1, Figure 10).
  4. The limitations are stated directly: HSR uses privileged state, scene drift remains under long intervals, and contact-rich tasks are left for future work. Algorithm 1 and hyperparameter Table 2 are provided.
Critical weaknesses
  1. As a benchmark, coverage is too narrow. All 16 tasks are the same pick-and-place template, with 2 objects and 2 receptacles per scene, and no contact-rich, deformable, tool-use, or articulated objects. Only OpenVLA-7B and SpatialVLA-4B are tested, and only with PPO. There is no newer VLA and no small from-scratch baseline to support the claim that VLA fine-tuning behaves differently.
  2. The strongest improvement does not transfer. HSR’s 14%→64% uses simulator ground truth; LSR and PER fail. The paper mainly shows that an oracle reset works and learned resets do not, which does not solve the real-world reset problem it poses.
  3. Comparisons are not normalized by reset budget, and Section 5 lacks a state-diversity-matched control, so the cause of the generalization gain is unclear.
  4. Statistical reporting is insufficient, reward and success are undefined, and the formulation does not match the algorithm.
Questions for authors
  1. How many scene resets do PR, HSR, HSR-only, LSR, and PER actually trigger during training? Please add a success-versus-reset-count curve at a matched reset budget.
  2. Is the reward sparse success or shaped and dense? What geometric threshold defines success?
  3. How many random seeds are behind Table 1 and Figure 10? Please report mean and standard deviation.
  4. If HSR uses only visual observations and no privileged state, how much does success drop?
4 comments

Weakness 1

As a benchmark, coverage is too narrow. All 16 tasks are the same pick-and-place template, with 2 objects and 2 receptacles per scene, and no contact-rich, deformable, tool-use, or articulated objects. Only OpenVLA-7B and SpatialVLA-4B are tested, and only with PPO.

Our evaluation scope is consistent with recent RL-for-VLA studies accepted at top venues, which also centre on OpenVLA and pick-and-place tasks [1, 2]. 

There is no small from-scratch baseline to support the claim that VLA fine-tuning behaves differently.

Our paper studies reinforcement learning fine-tuning of pre-trained robot foundation models under non-episodic settings. A from-scratch baseline would therefore address a different problem that falls outside the scope of this work.

References [1] J. Liu, F. Gao, B. Wei, X. Chen, Q. Liao, Y. Wu, C. Yu, and Y. Wang. What Can RL Bring to VLA Generalization? An Empirical Study. In Advances in Neural Information Processing Systems (NeurIPS), 2025.

[2] L. Shi, S. Chen, F. Gao, Y. Chen, K. Chen, T. Zhang, H. Zang, W. Zhang, C. Yu, and Y. Wang. Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Models. In Conference on Robot Learning (CoRL), 2026.

Weakness 2

CRONOS is primarily an analysis study of the limitations of non-episodic RL fine-tuning for pre-trained VLAs, rather than a method that solves the real-world reset problem. LSR and PER fail because their additional reset tasks are as sample-inefficient to learn as the original tasks; improving sample efficiency with model-based RL [1] is a promising direction.

We compared the ground-truth detector with an off-the-shelf VLM (Qwen3-VL-4B, zero-shot, without fine-tuning) on 4,608 frames covering all four scenes, with objects placed across a grid of table positions. The VLM achieves an accuracy of 0.903 and an F1 of 0.918, missing only 98 of 2,585 unrecoverable scenes (recall 0.962). Of these 98 misses, 93 are objects still on the table near the workspace boundary, and only 5 are objects that have fallen off the table. These results suggest that HSR does not need to rely on a privileged detector.

[1] Z. Yang, T. M. Moerland, M. Preuss, A. Plaat, and E. S. Hu. Reset-free reinforcement learning with world models. Transactions on Machine Learning Research, 2025.

Weakness 3

Section 5 lacks a state-diversity-matched control, so the cause of the generalization gain is unclear.

Non-episodic training exposes the policy to diverse yet temporally correlated object configurations, which can facilitate learning. In contrast, episodic training independently resamples object configurations at the start of each episode. As shown in the table below, episodic training with object positions uniformly sampled across the workspace underperforms non-episodic training under the evaluation setting of Section 5.1. We report results averaged over three seeds per method, each trained for 655K interaction steps.

MethodOverlap (%)Cluttered (%)
Episodic, full-workspace positions (T = 80)32.127.5
Episodic (T = 80)38.635.2
Non-episodic (T = 320)43.437.8

Weakness 4

formulation does not match the algorithm.

Following Gupta et al. [1], we formulate the task distribution p(·) as a general sampling scheme that is not restricted to resampling at every time step. We instead maintain each task for 80 steps to match the evaluation episode length, since changing the task objective at every step prevents effective policy rollout and learning.

Reference [1] A. Gupta, J. Yu, T. Z. Zhao, V. Kumar, A. Rovinsky, K. Xu, T. Devlin, and S. Levine. Reset-free reinforcement learning via multi-task learning: Learning dexterous manipulation behaviors without human intervention. In IEEE International Conference on Robotics and Automation (ICRA), pp. 6664–6

Question 1

How many scene resets do PR, HSR, HSR-only, LSR, and PER actually trigger during training? Please add a success-versus-reset-count curve at a matched reset budget.

Table A reports the success rate at matched reset counts, and Table B reports the number of scene resets triggered within the same 1.31M interaction steps. Resetting unrecoverable scenes does not add overhead to the reset budget, as HSR achieves a higher success rate for the same number of resets used.

Table A. Success at matched reset counts (in-domain success rate %).

# ResetsPRHSRHSR-onlyLSRPERCL
25011.2 ± 4.714.9 ± 3.311.2 ± 8.69.5 ± 8.612.1 ± 8.611.8 ± 2.3
50013.6 ± 6.216.4 ± 2.89.9 ± 8.212.7 ± 12.31.1 ± 0.010.4 ± 1.0
7508.2 ± 7.125.5 ± 2.720.8 ± 9.413.7 ± 13.72.1 ± 1.715.3 ± 0.7
100010.2 ± 5.747.0 ± 11.334.0 ± 23.41.2 ± 1.20.7 ± 0.220.2 ± 1.5

Table B. Reset cost at matched interaction steps (1.31M steps).

Method# Scene resetsHeuristic / PeriodicHeuristic trigger rateAvg. steps between resetsSuccess rate (%)
PR512 ± 00 / 512–2,56013.9 ± 7.2
HSR1,270 ± 122758 / 5124.6%1,04164.8 ± 2.0
HSR-only1,044 ± 62916 / 1285.6%1,26141.6 ± 31.4
LSR512 ± 00 / 512–2,56012.7 ± 12.3
PER512 ± 00 / 512–2,5601.0 ± 0.2
CL2,304 ± 00 / 2,304–56939.8 ± 12.5

Question 2

Is the reward sparse success or shaped and dense? What geometric threshold defines success?

Sparse rewards are given for grasping the object, maintaining the grasp, and successfully placing it on the target receptacle.

Success is defined by both geometric and contact thresholds. The object must be horizontally within half the receptacle's maximum bounding-box extent plus 1 cm, vertically within 5 cm of the receptacle surface, and exert a contact force greater than 0.03 N.

Question 3

How many random seeds are behind Table 1 and Figure 10? Please report mean and standard deviation.

Table 1 is averaged over 3 random seeds, and Figure 10 over 3 random seeds. We report mean ± standard deviation across seeds in Table D.

ModelTOverlapCluttered
OpenVLA800.554 ± 0.0590.494 ± 0.050
3200.593 ± 0.0020.536 ± 0.039
SpatialVLA800.552 ± 0.0970.512 ± 0.077
3200.643 ± 0.1350.555 ± 0.137

Seen task sequences :

ModelTTask 1Task 2Task 3Task 4
OpenVLA800.572 ± 0.0910.262 ± 0.0880.129 ± 0.0550.055 ± 0.039
3200.650 ± 0.0030.406 ± 0.0170.221 ± 0.0140.096 ± 0.008
SpatialVLA800.736 ± 0.0520.180 ± 0.1440.117 ± 0.1490.037 ± 0.052
3200.711 ± 0.1050.332 ± 0.2980.188 ± 0.2150.119 ± 0.157

Unseen task sequences :

ModelTTask 1Task 2Task 3Task 4
OpenVLA800.535 ± 0.0730.208 ± 0.0490.080 ± 0.0140.030 ± 0.011
3200.606 ± 0.0320.288 ± 0.0240.148 ± 0.0230.066 ± 0.011
SpatialVLA800.572 ± 0.1180.171 ± 0.1420.047 ± 0.0640.022 ± 0.031
3200.626 ± 0.1290.255 ± 0.2060.118 ± 0.1450.073 ± 0.096

Question 4

If HSR uses only visual observations and no privileged state, how much does success drop?

In our study, HSR is used to show that when to reset matters, not how to detect that a reset is needed. We therefore use privileged state as an oracle trigger, which isolates the effect of reset timing from errors in failure detection. Detecting unrecoverable states from visual observations is an orthogonal problem and we leave this to future work.

We thank the reviewer for the review. We have posted our response and updated the PDF. Please let us know if any concerns remain; we are happy to clarify.

We have updated the manuscript, with changes highlighted in blue in Table 1 and the appendix. These revisions also incorporate suggestions from the other reviewers.

Reviewer_DAyCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the problem of current RL based fine-tuning of VLAs where after every episode the scene resets and returns to the initial condition. But real world RL is not episodic, and physical scene reset is expensive. The authors introduced a simulation benchmark to study what happens when a robot continuously performs manipulation tasks without resetting the entire scene between tasks.

Critical strengths
  1. This paper opens a very useful investigation into how pretrained VLAs behave under infrequent resets and it is a worthwhile direction that is relatively underexplored.
  2. The benchmark supports controlled comparisons between different reset frequencies which provides reproducible setting for evaluating non-episodic performance of VLAs
Critical weaknesses
  1. One of the main motivations of the paper is to reduce human intervention. However, most comparisons focus on task success rather than reset costs, making it difficult to assess the trade-off between task performance and the cost of resetting the environment.
  2. The reported improvements are demonstrated only in simulation. The authors do not show how these improvements translate to real-world applications.
  3. CRONOS contains only four scenes and sixteen tasks, all involving tabletop manipulation. Therefore, the benchmark has limited diversity and task complexity.
  4. The benchmark's failure modes are primarily associated with objects becoming unreachable, which represents only one type of failure in non-episodic robotic learning.
Questions for authors
  1. Could the authors report how do the methods compare in terms of task success per reset or performance against a fixed intervention budget?
  2. Would an episodic baseline trained with a similarly diverse distribution of object configurations achieve similar generalization?
  3. How would the authors expect the observed findings to change with more objects, irreversible manipulations or tasks requiring recovery rather than another pick-and-place action?
  4. Equation 2 models task sampling and resets at individual time steps, while Algorithm 1 performs task switches every 80 steps and resets at task boundaries. Could the authors clarify the relationship between the formal objective and implementation?
5 comments

Weakness 1 and Question 1

One of the main motivations of the paper is to reduce human intervention. However, most comparisons focus on task success rather than reset costs, making it difficult to assess the trade-off between task performance and the cost of resetting the environment.

In Table A, we show the success rate at matched reset count, and in Table B, we report the number of resets each method triggers within the same 1.31M interaction steps. Resetting unrecoverable scenes does not add overhead to the reset budget, as HSR achieves a higher success rate for the same number of resets used.

Table A. Success at matched reset counts (in-domain success rate %).

# ResetsPRHSRHSR-onlyLSRPERCL
25011.2 ± 4.714.9 ± 3.311.2 ± 8.69.5 ± 8.612.1 ± 8.611.8 ± 2.3
50013.6 ± 6.216.4 ± 2.89.9 ± 8.212.7 ± 12.31.1 ± 0.010.4 ± 1.0
7508.2 ± 7.125.5 ± 2.720.8 ± 9.413.7 ± 13.72.1 ± 1.715.3 ± 0.7
100010.2 ± 5.747.0 ± 11.334.0 ± 23.41.2 ± 1.20.7 ± 0.220.2 ± 1.5

Table B. Reset cost at matched interaction steps (1.31M steps).

Method# Scene resetsHeuristic / PeriodicHeuristic trigger rateAvg. steps between resetsSuccess rate (%)
PR512 ± 00 / 512–2,56013.9 ± 7.2
HSR1,270 ± 122758 / 5124.6%1,04164.8 ± 2.0
HSR-only1,044 ± 62916 / 1285.6%1,26141.6 ± 31.4
LSR512 ± 00 / 512–2,56012.7 ± 12.3
PER512 ± 00 / 512–2,5601.0 ± 0.2
CL2,304 ± 00 / 2,304–56939.8 ± 12.5

Weakness 2

The reported improvements are demonstrated only in simulation.

We evaluate in simulation because, as noted in the paper, "experiments are primarily conducted in simulation benchmarks, as they are less time-consuming and offer better reproducibility than real-world benchmarks."

Our benchmark is built on SimplerEnv [1] and its extension [2], both of which have established strong sim-to-real alignment: SimplerEnv shows that policy performance in simulation correlates with real-world performance (Fig. 1 in [1]), and [2] further examines the sim-to-real generalization of trained policies. Notably, we do not modify the underlying physics engine or the 3D object assets, but introducing more complex scene compositions and more flexible reset mechanisms. The sim-to-real alignment established in prior work is therefore preserved, while our benchmark enables the study of more challenging and realistic interaction settings.

[1] Li, X., et al. Evaluating real-world robot manipulation policies in simulation. CoRL 2024. arXiv:2405.05941.

[2] Liu, J., et al. What can RL bring to VLA generalization? An empirical study. arXiv:2505.19789, 2025.

Weakness 3

CRONOS contains only four scenes and sixteen tasks, all involving tabletop manipulation.

We chose tabletop manipulation as a first study of reset-free RL for VLAs because it is a well-controlled setting with real-to-sim-aligned assets, where the effect of reset-free training can be isolated and measured reliably. Our benchmark supports 25 objects, 17 receptacles, and 21 table backgrounds, yielding 425 distinct pick-and-place tasks. In this work, we study non-episodic RL fine-tuning of pre-trained robot foundation models on a representative subset of these scenes.

Weakness 4

The benchmark's failure modes are primarily associated with objects becoming unreachable, which represents only one type of failure in non-episodic robotic learning.

Unreachable objects are only one of several failure modes in CRONOS. We distinguish training-time and test-time failures.

Training-time failure modes. Unreachable objects are the only failure that triggers an external reset, because the agent cannot recover from it by itself. The policy must handle all other non-episodic failures on its own: (1) State-distribution drift: objects are left clustered, stacked, or displaced, which we evaluate explicitly in the Overlap and Cluttered settings. (2) Error accumulation: a failed task leaves the scene in a state that lowers the success of the next one. (3) Robot-state drift: the arm can end a segment in an arbitrary pose.

Test-time failure modes. Our evaluation logs per-trial success, grasp, and chained success, which reveals several failures: (1) Grasp failure: the policy never achieves a stable grasp of the target object. (2) Placement failure: the object is grasped but does not end up resting on the target receptacle. (3) Object leaves the workspace: objects roll off the table or out of view. (4) Stacked or occluded objects in the Overlap and Cluttered settings. (5) Cascading failure: in sequential evaluation, success drops when the previous task failed .

Question 2

Would an episodic baseline trained with a similarly diverse distribution of object configurations achieve similar generalization?

Non-episodic training exposes the policy to diverse yet temporally correlated object configurations, which can facilitate learning. In contrast, episodic training independently resamples object configurations at the start of each episode. As shown in the table below, episodic training with object positions uniformly sampled across the workspace underperforms non-episodic training under the evaluation setting of Section 5.1. We report results averaged over three seeds per method, each trained for 655K interaction steps.

MethodOverlap (%)Cluttered (%)
Episodic, full-workspace positions (T = 80)32.127.5
Episodic (T = 80)38.635.2
Non-episodic (T = 320)43.437.8

Question 3

How would the authors expect the observed findings to change with more objects, irreversible manipulations or tasks requiring recovery rather than another pick-and-place action?

We further evaluate CRONOS in a larger scene with three objects and three receptacles. As shown in the table below, the periodic-reset baseline matches the episodic baseline at a fixed interaction budget, while achieving a 44% absolute gain in success rate at a fixed budget of 9,000 resets. This advantage in reset efficiency is even larger than in the 2×2 setting, which indicates that the trade-off between interaction efficiency and reset efficiency persists as task complexity increases, and that non-episodic learning remains the more reset-efficient choice as the task suite grows.

SettingTSuccess @ 3M interaction stepsSuccess @ 9,000 resets
3×3800.66 ± 0.100.18 ± 0.05
3200.63 ± 0.100.62 ± 0.10

For irreversible manipulations, the goal of non-episodic learning is to let the agent keep learning without resets, and irreversible tasks break this premise. For example, once a vegetable has been cut, the agent cannot restore it to its initial state, so it cannot practice the task again without external intervention. Not every task is suitable for non-episodic training, and we therefore consider such tasks outside the scope of our study. Tasks that require recovery, in contrast, fit naturally into our setting. Recovery can itself be treated as a task, so the agent learns to recover in the same way it learns any other task in CRONOS.

Question 4

Equation 2 models task sampling and resets at individual time steps, while Algorithm 1 performs task switches every 80 steps and resets at task boundaries. Could the authors clarify the relationship between the formal objective and implementation?

Following Gupta et al. [1], we formulate the task distribution p(·) as a general sampling scheme that is not restricted to resampling at every time step. We instead maintain each task for 80 steps to match the evaluation episode length, since changing the task objective at every step prevents effective policy rollout and learning.

Reference [1] A. Gupta, J. Yu, T. Z. Zhao, V. Kumar, A. Rovinsky, K. Xu, T. Devlin, and S. Levine. Reset-free reinforcement learning via multi-task learning: Learning dexterous manipulation behaviors without human intervention. In IEEE International Conference on Robotics and Automation (ICRA), pp. 6664–6

Thank you for the detailed rebuttal and additional experiments.

The matched-reset comparisons, stronger episodic baseline, and 3×3 scene evaluation substantially address my concerns regarding reset efficiency and the generalizability of the findings.

The sim-to-real argument does not directly answer the real-world validity of the proposed setting, the evaluated task diversity is still limited and recovery from failures is discussed but not experimentally shown.

Reviewer_oBQGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CRONOS, a simulation benchmark for reinforcement-learning fine-tuning of VLAs under infrequent scene resets. It contains 16 language-conditioned pick-and-place tasks across four shared scenes, with tasks switching every 80 steps and scenes reset at intervals of 80, 320, or 2560 steps.

Critical strengths
  • The paper targets a practically important bottleneck in robot RL: frequent manual scene resets. Sharing scenes across tasks allows object displacement and failures to accumulate across task boundaries, making the setup more realistic than evaluating every task from an independently reset scene.
  • The study includes two sizeable pretrained VLAs and reports informative negative results, including the failure of naive training at very infrequent resets and the poor performance of learned/perturbation reset variants. These observations could guide future work on autonomous VLA adaptation.
Critical weaknesses
  • The paper never measures its central quantity, reset efficiency. It reports nominal reset intervals but not the actual scene-reset counts, heuristic-trigger rates, intervention time, or autonomous operating duration for each method. Curriculum learning uses more resets early, and heuristic resets add state-dependent resets.
  • The statistical evidence is inadequate for the main conclusions. The manuscript does not state the number of training seeds or evaluation trials, does not define the shaded regions, and provides no uncertainty for the OOD table or sequence figure.
  • The benchmark is narrow and under-specified. It contains only 16 pick-and-place tasks in four scenes and one simulator, while robot embodiment, cameras, action/control details, rewards, success predicates, assets, and evaluation sampling are incomplete.
Questions for authors

see above

5 comments

Weakness 1

The paper never measures its central quantity, reset efficiency. It reports nominal reset intervals but not the actual scene-reset counts, heuristic-trigger rates, intervention time, or autonomous operating duration for each method. Curriculum learning uses more resets early, and heuristic resets add state-dependent resets.

In Table A, we show the success rate at matched reset count. HSR turns each reset into the most learning progress. Human intervention time cannot be measured in simulation, so we count each scene reset as one human intervention. In Table B, we report the number of resets each method triggers within the same 1.31M interaction steps.

HSR thus requests intervention 2.5× as often as PR but achieves 4.7× higher success (64.8% vs. 13.9%), and it outperforms CL (64.8% vs. 39.8%) with 45% fewer resets. A good reset algorithm should therefore not simply minimize interventions, but spend them on scenes that have become unrecoverable.

Table A. Success at matched reset counts (in-domain success rate %).

# ResetsPRHSRHSR-onlyLSRPERCL
25011.2 ± 4.714.9 ± 3.311.2 ± 8.69.5 ± 8.612.1 ± 8.611.8 ± 2.3
50013.6 ± 6.216.4 ± 2.89.9 ± 8.212.7 ± 12.31.1 ± 0.010.4 ± 1.0
7508.2 ± 7.125.5 ± 2.720.8 ± 9.413.7 ± 13.72.1 ± 1.715.3 ± 0.7
100010.2 ± 5.747.0 ± 11.334.0 ± 23.41.2 ± 1.20.7 ± 0.220.2 ± 1.5

Table B. Reset cost at matched interaction steps (1.31M steps).

Method# Scene resetsHeuristic / PeriodicHeuristic trigger rateAvg. steps between resetsSuccess rate (%)
PR512 ± 00 / 512–2,56013.9 ± 7.2
HSR1,270 ± 122758 / 5124.6%1,04164.8 ± 2.0
HSR-only1,044 ± 62916 / 1285.6%1,26141.6 ± 31.4
LSR512 ± 00 / 512–2,56012.7 ± 12.3
PER512 ± 00 / 512–2,5601.0 ± 0.2
CL2,304 ± 00 / 2,304–56939.8 ± 12.5

Weakness 2

The statistical evidence is inadequate for the main conclusions. The manuscript does not state the number of training seeds or evaluation trials, does not define the shaded regions, and provides no uncertainty for the OOD table or sequence figure.

All training curves are averaged over 3 random seeds, and the shaded regions denote standard deviation across seeds. For evaluation, each checkpoint is evaluated on 1,024 trials per setting, and we report the mean ± std across seeds. In the OOD evaluation, non-episodic resetting (T=320) achieves a higher mean success rate than episodic resetting (T=80) in all settings.

ModelTOverlapCluttered
OpenVLA800.554 ± 0.0590.494 ± 0.050
3200.593 ± 0.0020.536 ± 0.039
SpatialVLA800.552 ± 0.0970.512 ± 0.077
3200.643 ± 0.1350.555 ± 0.137

For the sequential evaluation, the tables below report the chained success rate (a task counts as successful only if all preceding tasks in the sequence also succeeded) as mean ± std across seeds. T=320 achieves a higher mean chained success than T=80 on every task after the first, for both models and for both seen and unseen task sequences, showing that NSR-MTRL improves robustness in long-horizon robotic manipulation.

Seen task sequences :

ModelTTask 1Task 2Task 3Task 4
OpenVLA800.572 ± 0.0910.262 ± 0.0880.129 ± 0.0550.055 ± 0.039
3200.650 ± 0.0030.406 ± 0.0170.221 ± 0.0140.096 ± 0.008
SpatialVLA800.736 ± 0.0520.180 ± 0.1440.117 ± 0.1490.037 ± 0.052
3200.711 ± 0.1050.332 ± 0.2980.188 ± 0.2150.119 ± 0.157

Unseen task sequences :

ModelTTask 1Task 2Task 3Task 4
OpenVLA800.535 ± 0.0730.208 ± 0.0490.080 ± 0.0140.030 ± 0.011
3200.606 ± 0.0320.288 ± 0.0240.148 ± 0.0230.066 ± 0.011
SpatialVLA800.572 ± 0.1180.171 ± 0.1420.047 ± 0.0640.022 ± 0.031
3200.626 ± 0.1290.255 ± 0.2060.118 ± 0.1450.073 ± 0.096

Weakness 3

The benchmark is narrow and under-specified. It contains only 16 pick-and-place tasks in four scenes and one simulator, while robot embodiment, cameras, action/control details, rewards, success predicates, assets, and evaluation sampling are incomplete.

Our simulation benchmark supports more tasks than those used in the paper. It contains 25 objects, 17 receptacles, and 21 table backgrounds, giving 425 distinct pick-and-place tasks. Because we focus on studying non-episodic scene reset, our experiments were conducted on 16 out of 425 pick-and-place tasks. We commit to releasing the code base, allowing users to train on all 425 tasks via NSR-MTRL. To enhance your understanding of our benchmark, we show a more complet benchmark configuration as follows:

CategoryItemSpecification
PlatformSimulatorManiSkill3
RobotWidowX 250S, BridgeData V2 digital twin from SimplerEnv
CameraSingle third-person RGB, 640 × 480; no wrist camera
Action space7-D: EE position delta (3), rotation delta (3), binary gripper (1)
AssetsObjects25
Receptacles17
Table backgrounds21
TasksTask familyPick-and-place: "put <object> on <receptacle>"
Distinct tasks425
Distinct scenes (2 × 2)856,800
Per-scene randomization432 collision-free layouts × 4 object orientations
EvaluationRewardSparse staged reward: +0.1 target object grasped, +0.1 grasp held for 5 consecutive steps, +1.0 object on the target receptacle while grasped
Success predicateObject on the target receptacle: xy distance within the receptacle's half bounding-box diagonal (+3 mm), object above the receptacle with a vertical gap ≤ 2 cm
Evaluation sampling64 parallel environments (16 per scene) × 4 evaluation episodes = 256 trials, 16 per task. Each episode starts from a full reset and lasts 80 control steps.

Thank you for providing the reset-efficiency tables, seed/uncertainty statistics, and benchmark specifications. These additions clarify the experimental setup, though I note they have not yet been incorporated into the uploaded PDF.

Before finalizing my evaluation, could you briefly clarify three points raised by the new results?

  1. W1 (Reset Accounting & Episodic Baseline): In Table B, PR/LSR/PER trigger only 512 resets over 1.31M steps, yet Table A reports them at 750 and 1,000 resets—were they trained for ~2.56M steps in Table A (comparing unequal interaction budgets)? Also, why does HSR-only still list 128 periodic resets (916 / 128), and how does the episodic baseline (T=80T=80) perform at matched reset counts in Table A?
  2. W2 (Trial-Count Discrepancy & High Seed Variance): The response to W2 states 1,024 trials per setting, whereas W3 states 64 environments × 4 episodes = 256 trials (and SpatialVLA Cluttered is 0.550 in Table 1 vs. 0.555 in the rebuttal). Could you clarify this discrepancy? Moreover, across 3 seeds, several SpatialVLA standard deviations remain very large relative to the claimed gains (e.g., ±0.135–0.137 in OOD, 0.332 ± 0.298 and 0.073 ± 0.096 in sequential evaluation, and 41.6 ± 31.4% for HSR-only), making parts of the SpatialVLA comparison statistically noisy.
  3. W3 (Task Scope & Reward/Success Predicate): Even with 425 object–receptacle pairings, the benchmark covers a single "put <object> on <receptacle>" tabletop primitive (with all experiments still on 16 tasks). In addition, the specification table defines the +1.0 reward as object on target "while grasped" and omits a release/static check in the success predicate—is "while grasped" a typo, or can a trial succeed without releasing the object before task switch / EER?

Clarifying these points (and updating the manuscript accordingly) will be very helpful for my final assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Revisions

We thank the AC and the reviewers (oBQG, DAyC, r4P1) for their time. To facilitate the discussion, we summarize the reviewers' assessment, our responses, and the revisions below.

Strengths. All reviewers agree that the direction is important: understanding how pretrained VLAs behave when fine-tuned with RL under infrequent scene resets, a practical bottleneck for real-world RL that remains underexplored. Reviewers also value the clear failure diagnosis (r4P1), the informative negative results (oBQG), and the generalization gains of non-episodic fine-tuning (r4P1).

How we addressed the concerns.

  1. Experimental details and statistics (oBQG, r4P1). We provide the complete benchmark specification, including the reward, success predicate, and evaluation sampling (Appendix E), and report detailed statistics, i.e., mean ± std across seeds, for the main results. Some SpatialVLA results show high variance across seeds, which we attribute to error accumulation in long-horizon non-episodic training.

  2. Reset efficiency (oBQG, DAyC, r4P1). We report success at matched reset counts and the number of resets at matched interaction steps (Appendix D). At 1,000 resets, HSR reaches 47.0% success, versus 10.2% for periodic resets, 20.2% for curriculum learning, and 14.9% for an episodic baseline.

  3. Additional experiments (oBQG, DAyC, r4P1).

    • We scale up to larger scenes with three objects and three receptacles, where non-episodic training remains more reset-efficient.
    • A state-diversity-matched episodic control still underperforms non-episodic training (Appendix G), suggesting that the generalization gain does not stem from diversity alone.
    • To extend task diversity beyond pick-and-place, we add a block-stacking task, where each task serves as the recovery of the other. The benefits of non-episodic training still hold, and when starting from a reversed stack, it reaches 52.7% and 56.6% success versus 8.4% and 5.3% for episodic training, providing experimental evidence of recovery.
  4. Transfer to the real world without privileged state (r4P1, DAyC). Every reset mechanism we study can be implemented with signals available on a real robot. For HSR, an off-the-shelf VLM detects unrecoverable scenes from camera images with an F1 of 0.918 against the ground-truth rule.

Limitation. We do not include real-robot experiments or closed-loop training with the VLM-based detector, which we leave to future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.