acceptodds
Under review as a conference paper at ICLR 2027

Predicting Irreversible Transitions for Safe Switching in Autonomous RL

18 views

Abstract

Autonomous reinforcement learning (ARL) aims to reduce manual resets by alternating between forward and reset policies. One of the main challenges in ARL is handling irreversible states, from which the agent cannot recover without external intervention. While several recent works have proposed ARL algorithms that can handle irreversible states, these algorithms require task-specific knowledge, such as privileged state information or reversibility labels. Furthermore, they pay little attention to safely switching between forward and reset policies. In this paper, we propose a novel ARL algorithm that identifies irreversible states from image observations without task-specific knowledge and enables safe policy switching. Our algorithm introduces an irreversibility estimator that takes the current image observation and action as input and estimates the probability of transitioning to an irreversible state. We train the estimator on the state-action sequences that lead to irreversible states, so that it predicts an irreversible transition several steps before it occurs and aborts the ongoing policy early. The supervisory signals for these sequences are extracted from rollouts using vision-language models. Our algorithm also introduces a transition policy that guides the agent to safe switching states. These states are reversible, not too difficult for resuming the forward policy, and reachable from the aborted state. Experimental results demonstrate that our algorithm outperforms prior ARL algorithms across diverse navigation and manipulation tasks, even without task-specific knowledge.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Autonomous Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7ScrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of reducing manual resets in autonomous reinforcement learning (ARL) with irreversible states. The authors argue that detecting irreversible states only when the agent is already close to them may leave insufficient time for recovery, while directly switching policies after early termination may expose the agent to further risks. To address these issues, the paper proposes a multi-step irreversibility estimator and a transition policy. The estimator uses VLM-generated labels to predict potential irreversible transitions several steps in advance, while the transition policy guides the agent toward suitable states for resuming learning. Experiments on four navigation and manipulation environments show that the proposed method reduces manual resets while maintaining competitive task success rates.

Critical strengths
  1. The paper addresses a practical problem in ARL. I find the motivation reasonable, particularly the observation that early termination alone may not be sufficient if the agent cannot safely resume learning from the aborted state.
  2. The paper combines multi-step irreversibility prediction with a transition policy to address both early risk detection and safe policy switching. The experimental results show clear reductions in manual resets across four environments, and the ablation studies provide evidence that both components contribute to these improvements.
Critical weaknesses
  1. Limited comparison with related work and insufficient validation of multi-step irreversibility prediction. The paper lacks a comparison with Robust and Scalable Autonomous Reinforcement Learning in Irreversible Environments (NeurIPS 2025), which addresses a similar problem and shares some underlying mechanisms and experimental settings. A direct comparison would help clarify the advantages of the proposed approach. In addition, although multi-step irreversibility prediction is well motivated, the N-step label expansion relies on future trajectory outcomes that also depend on subsequent actions, raising questions about how accurately the estimator captures action-dependent risks. Fig. 5 mainly provides qualitative results, without a sensitivity analysis of N or quantitative evaluation of prediction lead time and false positive rates. Further experimental evidence would help establish the effectiveness of the proposed multi-step prediction mechanism.
  2. The safety of the transition policy is not sufficiently established. Some aspects of the proposed framework require further justification. First, Eq. (8) excludes states with high estimated reachability, although such states may be preferable for safe recovery. Moreover, Algorithm 2 does not explicitly check irreversibility risks during transition execution, and the current experiments provide limited evidence of transition safety. Second, although the paper claims to avoid task-specific knowledge, Appendix B.2 shows that the VLM is fine-tuned separately for each environment using labeled reversible and irreversible images. While eliminating online human annotation is valuable, the method still relies on environment-specific supervision. Further clarification of these design choices and supervision requirements would strengthen the paper's claims regarding safety and real-world applicability.
Questions for authors
  1. Could the authors provide a direct comparison with Robust and Scalable Autonomous Reinforcement Learning in Irreversible Environments (NeurIPS 2025), at least on the shared environments? This would help clarify the benefits of the proposed irreversibility estimator and transition policy over existing approaches.
  2. What value of N is used in the experiments, and how sensitive is the method to this choice? Fig. 5 shows the effect of introducing N-step expansion, but does not examine different values of N. A comparison of prediction lead time, false positive rates, manual resets, and success rates under different settings would be useful.
  3. Why does Eq. (8) exclude states with high estimated reachability? Also, how are potentially irreversible actions handled during the execution of the transition policy? It would be helpful to report the transition success rate and the frequency of irreversible events during these transitions, as well as clarify what happens when the selected switching state cannot be reached.
6 comments

Thank you for carefully reading our paper and for your specific comments. In particular, your comments on the comparison with RSA, the sensitivity to NN, the safety of the transition policy, and the environment-specific supervision helped us present our contributions more clearly. We have revised the paper to address all of your comments and marked all changes in blue.


[W1-1 & Q1] Direct comparison with RSA

The direct comparison you suggested allowed us to show the benefits of our irreversibility estimator and transition policy over RSA. To address your concern, we added RSA as a baseline in all four environments, including PointMaze-Trap and AntOpen-Trap, which are shared with the RSA paper. We also included RSA in a controlled comparison in which every condition uses the same curriculum with the state information estimator (Table 12 of Appendix F.2). For a fair comparison, RSA is trained with the same hyperparameters as our algorithm.

In the shared environments, our algorithm requires 30.6% fewer manual resets than RSA in PointMaze-Trap (184 vs. 265) and 52.1% fewer in AntOpen-Trap (68 vs. 142), with equal or higher success rates. In all four environments, it requires at least 29.7% fewer manual resets than RSA (Table 16 of Appendix F.6). In the controlled comparison, our irreversibility estimator alone, without the transition policy, requires at least 17.4% fewer manual resets than RSA, and adding the transition policy reduces them by at least a further 11.8%.

EnvironmentMetricRSAEstimator (N=10N=10)Ours
PointMaze-TrapSuccess rate (%)100.0100.0100.0
Manual resets265219181
AntOpen-TrapSuccess rate (%)88.389.291.7
Manual resets1429763
Fetch-Reach-TrapSuccess rate (%)79.284.287.5
Manual resets1,105865455
Highway-Parking-TrapSuccess rate (%)81.382.585.0
Manual resets1,190960847

Success rates (%) and manual resets, mean over five random seeds (Table 12 of Appendix F.2). Estimator (N=10N=10) is our irreversibility estimator without the transition policy. The fewest manual resets are in bold.

These results show that each of our two contributions reduces manual resets. Unlike RSA, which passes state inputs through a mapping function and uses successor features to detect irreversible actions, our irreversibility estimator requires no mapping function and predicts an irreversible transition NN steps in advance from the image observation and action. The transition policy then guides the agent to a safe switching state before the next policy starts. Table 12 is newly added in Appendix F.2, and RSA is newly added as a baseline in Figure 4 and Table 16 of Appendix F.6.

[W1-2 & Q2] Validation of multi-step prediction

Your comments led us to validate the multi-step prediction quantitatively. To address your concern, we added a quantitative evaluation of the irreversibility estimator (Table 13 of Appendix F.3) and a sensitivity analysis for N∈{0,5,10,20}N\in\{0,5,10,20\} (Figure 11 and Table 14 of Appendix F.4), and we now state that N=10N=10 is used in all environments (Table 2 of Appendix B.1). We freeze the trained irreversibility estimator, collect rollouts without early aborts, and use as the target whether each (ot,at)(o_t,a_t) is followed by an irreversible transition within NN steps. The lead time is the median number of steps from the first warning prediction to the irreversible transition, and the false positive rate (FPR) is the fraction of (ot,at)(o_t,a_t) pairs not followed by an irreversible transition within NN steps that receive a warning prediction.

With N=10N=10, the irreversibility estimator issues warnings at least 7 steps before the irreversible transition, compared with 1 step without NN-step expansion. The AUROC is at least 0.91, and at τwarning\tau_{\mathrm{warning}} the FPR is at most 5.2%.

MetricPointMaze-TrapAntOpen-TrapFetch-Reach-TrapHighway-Parking-Trap
Lead time (steps), N=0N=01111
Lead time (steps), N=10N=108879
FPR (%)1.11.40.95.2
AUROC0.970.950.980.91

Evaluation of the irreversibility estimator in the four environments (Table 13 of Appendix F.3). The columns other than the lead time are for N=10N=10.

As NN increases from 0 to 20, the lead time grows from 1 step to at least 15 steps, and the FPR also rises. Increasing NN from 0 to 10 reduces manual resets in both environments, while the success rate stays at 100.0% in PointMaze-Trap and rises in Highway-Parking-Trap. Increasing NN from 10 to 20 reduces manual resets by no more than a further 5.3% and lowers the success rate in Highway-Parking-Trap.

EnvironmentNNLead time (steps)FPR (%)Manual resetsSuccess rate (%)
PointMaze-Trap010.3226100.0
540.6203100.0
1081.1181100.0
20152.417698.3
Highway-Parking-Trap011.01,01283.8
542.493185.4
1095.284785.0
201610.880276.3

Results for different values of NN (Table 14 of Appendix F.4).

These results show that the multi-step prediction issues accurate warnings several steps before the irreversible transition and distinguishes the (ot,at)(o_t,a_t) pairs that lead to irreversible transitions, which captures action-dependent risks. A larger NN issues warnings earlier and reduces manual resets, but the additional reduction beyond N=10N=10 is small, so we use N=10N=10. Table 13 is newly added in Appendix F.3, and Figure 11 and Table 14 are newly added in Appendix F.4.


[W2-1 & Q3-1] Upper threshold in Eq. (8)

Your question helped us explain the role of the upper threshold in Eq. (8). We would like to note that the upper threshold λhigh\lambda_{\mathrm{high}} excludes candidates that are too close to the aborted state. An early abort occurs when the agent approaches an irreversible state, so candidates with very high estimated reachability from the aborted state usually lie within a few steps of it and are still close to the irreversible state.

To verify this claim, we conducted an additional experiment that removes λhigh\lambda_{\mathrm{high}}. Removing λhigh\lambda_{\mathrm{high}} increases manual resets from 181 to 232 in PointMaze-Trap and from 455 to 618 in Fetch-Reach-Trap (Table 15 of Appendix F.5).

Manual resetsPointMaze-TrapFetch-Reach-Trap
Ours181455
w/o λ_high232618

Effect of removing the upper threshold λ_high (Table 15 of Appendix F.5). The fewer manual resets are in bold.

These results show that resuming training next to the aborted state easily leads the agent into irreversible states again, and that λhigh\lambda_{\mathrm{high}} provides a margin from the aborted state. Table 15 is newly added in Appendix F.5.

[W2-2 & Q3-2] Irreversibility risks during transition execution

Your question led us to measure the safety of the transition policy directly. To address your concern, we added the transition success rate and the frequency of entries into irreversible states during transitions (Table 15 of Appendix F.5), and we now describe in Appendix A and Algorithm 2 how the transition policy is executed, including the case in which the selected state is not reached.

The transition policy does not evaluate pirrp_{\mathrm{irr}} during execution. The transition starts from the aborted state, where the predicted risk is high, so an early abort could stop the transition before the agent moves away from that state. Instead, the transition policy moves the agent toward the selected safe switching state, and an entry into an irreversible state during a transition is counted as a manual reset. If the agent does not reach the selected state within TtransitionT_{\mathrm{transition}} steps, training resumes from the state where the transition ends (line 11 of Algorithm 2). The next policy evaluates pirrp_{\mathrm{irr}} at every step from that state, so an early abort is triggered again if the agent selects a risky action.

The transition policy reaches the selected safe switching state in at least 85.7% of transitions in all four environments (89.4% on average), and entries into irreversible states during transitions account for at most 4.9% of all manual resets (3.9% on average).

EnvironmentReached the selected state (%)Manual resets during transitions (%)
PointMaze-Trap94.12.8
AntOpen-Trap86.54.9
Fetch-Reach-Trap91.23.1
Highway-Parking-Trap85.74.6

Transition policy statistics (Table 15 of Appendix F.5).

These results show that, even though the transition policy does not evaluate pirrp_{\mathrm{irr}} during execution, most transitions reach the selected safe switching state and only a small fraction of manual resets occurs during transitions. Table 15 is newly added in Appendix F.5.

Reviewer_5eQLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of handling irreversible states in autonomous reinforcement learning (ARL), where an agent alternates between forward and reset policies to reduce the need for manual interventions. The authors propose a framework that integrates two main components: an irreversibility estimator and a transition policy. The proposed method is evaluated in four simulated environments, covering navigation, robotic manipulation, and autonomous parking tasks.

Critical strengths
  1. The integration of anticipatory irreversibility prediction and safe policy switching is intuitively appealing.

  2. The experimental evaluation covers multiple tasks and includes useful ablation studies.

Critical weaknesses
  1. The claim of eliminating task-specific knowledge is not sufficiently supported by the experimental setup. A central claim of this paper is that the proposed method identifies irreversible states without task-specific knowledge. However, Appendix B.2 states that separate Qwen3-VL models are fine-tuned for each environment using simulator-generated reversibility labels. Table 3 reports 2,000 reversible and 2,000 irreversible training images for each environment. Furthermore, the task-agnostic prompts in Appendix D.2 still require environment-specific visual information, including the controllable agent and the visual appearance of irreversible regions. The task-specific prompts contain even more detailed geometric and dynamical information.

  2. Important baseline comparisons are missing, making it difficult to assess the methodological contribution. The paper compares its approach with LNT, PAINT, R3L, and their multi-goal variants. However, the authors cite Robust and Scalable Autonomous Reinforcement Learning in Irreversible Environments (Lee, NeurIPS 2025), which proposes RSA for handling irreversible states without task-specific knowledge, yet do not include RSA in the experimental comparisons.

  3. The effectiveness of multi-step irreversibility prediction is not quantitatively established. The proposed irreversibility estimator is trained by assigning positive labels to the first irreversible transition and the preceding N transitions. While this is a straightforward way to encourage early detection, it does not necessarily establish that the estimator accurately predicts the probability of entering an irreversible state within N future steps.

Questions for authors

See weaknesses.

3 comments

Thank you for carefully reading our paper and for your specific comments. In particular, your comments on the claim about task-specific knowledge, the comparison with RSA, and the quantitative validation of multi-step irreversibility prediction helped us present our contributions more clearly. We have revised the paper to address all of your comments and marked all changes in blue.


[W1-1] Fine-tuning a VLM for each environment

Your close reading of Appendix B.2 and Table 3 helped us describe what our algorithm requires more precisely. We would like to note that our algorithm can work with a VLM without environment-specific fine-tuning, and the labeled images in Table 3 are therefore optional. The only reason we use a small fine-tuned VLM is labeling efficiency during ARL training. Thus, we believe that our claim of identifying irreversible states without task-specific knowledge holds. Note that in our algorithm, the VLM is used only to provide a binary reversibility label for each image observation.

To verify this claim, we conducted additional experiments that compare our algorithm with the fine-tuned VLM and with a closed-source VLM without fine-tuning. We first compare the labeling performance of the two VLMs on the 2,000 held-out images. Without fine-tuning, GPT-6.1 Astra reaches an accuracy of at least 98.6% and an irreversible-state recall of at least 96.8% in all four environments, which is comparable to the fine-tuned Qwen3-VL (Table 8 of Appendix F.1).

EnvironmentMetricFine-tuned Qwen3-VLGPT-6.1 Astra
PointMaze-TrapAccuracy (%)99.1599.2
Recall (%)95.0297.6
AntOpen-TrapAccuracy (%)99.7098.8
Recall (%)99.7797.1
Fetch-Reach-TrapAccuracy (%)100.0099.4
Recall (%)100.0098.5
Highway-Parking-TrapAccuracy (%)98.7098.6
Recall (%)98.9896.8

Labeling performance on the 2,000 held-out images (Table 8 of Appendix F.1).

We then replace the fine-tuned VLM with GPT-6.1 Astra in our algorithm. This increases manual resets by no more than 4.4% in all four environments, with similar success rates (Table 10 of Appendix F.1). These results support our claim that the proposed algorithm does not need environment-specific fine-tuning.

EnvironmentMetricFine-tuned Qwen3-VLGPT-6.1 Astra
PointMaze-TrapSuccess rate (%)100.0100.0
Manual resets184186
AntOpen-TrapSuccess rate (%)91.790.0
Manual resets6871
Fetch-Reach-TrapSuccess rate (%)88.387.5
Manual resets467474
Highway-Parking-TrapSuccess rate (%)90.088.8
Manual resets837859

Success rates and manual resets of our algorithm with each VLM, mean over five random seeds (Table 10 of Appendix F.1).

In addition, the labeled images in Table 3 are needed only when the fine-tuned VLM is used; with GPT-6.1 Astra, no labeled images are needed. The table below reports the time per query for labeling an image (Table 9 of Appendix F.1); GPT-6.1 Astra takes more than 37 times longer per query than the fine-tuned Qwen3-VL.

ModelTime per query (ms)
Qwen3-VL-2B, fine-tuned25
Qwen3-VL-4B, fine-tuned40
GPT-6.1 Astra, zero-shot1,500

Overall, our algorithm does not need any environment-specific supervision, such as fine-tuning data. Indeed, our additional experiments show comparable labeling accuracy and task performance even without a fine-tuned VLM. Tables 8, 9, and 10 are newly added in Appendix F.1.

[W1-2] Environment-specific information in the prompts

We appreciate your careful reading of the prompts in Appendix D.2. We would like to note that the noun phrase for the irreversible region comes from how our simulated environments are built, not from a requirement of our algorithm. Our environments mark irreversible regions with colors defined by each environment, whose meaning cannot be inferred from the image alone. In the real world, however, many irreversible states are apparent from the image itself, such as a crashed car in autonomous driving or a fallen or damaged humanoid robot. We therefore expect that a VLM can identify such states with its general knowledge, without any environment-specific noun phrase. We believe that this direction is an important step toward reinforcement learning in the real world. Unlike prior ARL algorithms, which require task-specific knowledge such as the reset reward function in LNT and the reversibility labels from a human in PAINT (Appendix C), our algorithm leaves the identification of irreversible states to the general knowledge of a VLM. As a first step, a VLM without fine-tuning, given only the two noun phrases, already achieves success rates and manual resets comparable to those of the fine-tuned VLM (Table 10 of Appendix F.1).

EnvironmentMetricFine-tuned Qwen3-VLGPT-6.1 Astra
PointMaze-TrapSuccess rate (%)100.0100.0
Manual resets184186
AntOpen-TrapSuccess rate (%)91.790.0
Manual resets6871
Fetch-Reach-TrapSuccess rate (%)88.387.5
Manual resets467474
Highway-Parking-TrapSuccess rate (%)90.088.8
Manual resets837859

Success rates and manual resets of our algorithm with each VLM, mean over five random seeds (Table 10 of Appendix F.1).

We added the table above to Appendix F.1 as Table 10.


[W2] Comparison with RSA

We agree that RSA is an important baseline, and your comment led us to compare our algorithm directly with the most closely related prior algorithm. To address your concern, we added RSA as a baseline in all four environments. We also included RSA in a controlled comparison in which every condition uses the same curriculum with the state information estimator (Table 12 of Appendix F.2).

Our algorithm requires at least 29.7% fewer manual resets than RSA in all four environments, with an equal success rate in PointMaze-Trap and higher success rates in the other three environments (Table 16 of Appendix F.6). In the controlled comparison, our irreversibility estimator alone, without the transition policy, requires at least 17.4% fewer manual resets than RSA, and adding the transition policy reduces them by at least a further 11.8%.

EnvironmentMetricRSAEstimator (N=10N=10)Ours
PointMaze-TrapSuccess rate (%)100.0100.0100.0
Manual resets265219181
AntOpen-TrapSuccess rate (%)88.389.291.7
Manual resets1429763
Fetch-Reach-TrapSuccess rate (%)79.284.287.5
Manual resets1,105865455
Highway-Parking-TrapSuccess rate (%)81.382.585.0
Manual resets1,190960847

Success rates (%) and manual resets, mean over five random seeds (Table 12 of Appendix F.2). Estimator (N=10N=10) is our irreversibility estimator without the transition policy. The fewest manual resets are in bold.

These results show that each of our two contributions reduces manual resets. Unlike RSA, which passes state inputs through a mapping function and uses successor features to detect irreversible actions, our irreversibility estimator requires no mapping function and predicts an irreversible transition NN steps in advance from the image observation and action. The transition policy then guides the agent to a safe switching state before the next policy starts. Table 12 is newly added in Appendix F.2, and RSA is newly added as a baseline in Figure 4 and Table 16 of Appendix F.6.

[W3] Quantitative validation of multi-step irreversibility prediction

Your comment led us to evaluate quantitatively whether pirrp_{\mathrm{irr}} predicts an irreversible transition within NN steps. To address your concern, we added a quantitative evaluation of the irreversibility estimator (Table 13 of Appendix F.3). We freeze the trained irreversibility estimator, collect rollouts without early aborts, and use as the target whether each (ot,at)(o_t,a_t) is followed by an irreversible transition within NN steps.

The expected calibration error (ECE) between pirrp_{\mathrm{irr}} and the observed frequency is at most 0.068, and the AUROC is at least 0.91. At τwarning\tau_{\mathrm{warning}}, the false positive rate (FPR) is at most 5.2%. NN-step expansion increases the lead time from 1 step to at least 7 steps and, under the same curriculum, reduces manual resets by at least 10.7% relative to N=0N=0 (Table 12 of Appendix F.2).

MetricPointMaze-TrapAntOpen-TrapFetch-Reach-TrapHighway-Parking-Trap
Lead time (steps), N=0N=01111
Lead time (steps), N=10N=108879
FPR (%)1.11.40.95.2
AUROC0.970.950.980.91
ECE0.0310.0440.0270.068

Evaluation of the irreversibility estimator in the four environments (Table 13 of Appendix F.3). The columns other than the lead time are for N=10N=10.

These results show that pirrp_{\mathrm{irr}} follows the probability of entering an irreversible state within NN steps, and that NN-step expansion lets the irreversibility estimator issue warnings several steps before the irreversible transition. Tables 12 and 13 are newly added in Appendices F.2 and F.3, respectively.


We appreciate your time and effort in reviewing our work. If our response has addressed your concerns, we would greatly appreciate it if you could reconsider your score. We are also happy to discuss any further questions.

Reviewer_Ge4MOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work proposes an autonomous RL algorithm that lets an agent learn with fewer manual resets in environments where some states can't be recovered from. It contributes (1) an irreversibility estimator that looks at the current image and action and predicts whether the agent will enter an irreversible state within the next few steps, trained on labels from a VLM, and (2) a transition policy that moves the agent to a safe state after an early abort before training resumes. Experiments on four simulated navigation and manipulation tasks show comparable success rates to prior ARL methods with fewer manual resets.

Critical strengths

None

Critical weaknesses
  1. Claims exceed the demonstrated supervision requirements. The paper claims to identify irreversible states without task-specific knowledge, but the VLM is fine-tuned separately for each environment using 4,000 simulator-labeled images (App. B.2, Table 3). Without adaptation, irreversible-state recall is only 18.9–31.7% (Table 4). Appendix C acknowledges this offline supervision, so the supported claim is that the method avoids ongoing human annotation during ARL training—not environment-specific supervision altogether. Performance claims also need qualification: success rates are comparable to the strongest multi-goal baselines in three environments and lower than PAINT-MG and R3L-MG in Highway-Parking, although the method reports fewer resets.
  2. The real-world motivation is not sufficiently validated. All experiments use simulated environments with explicitly designated irreversible regions. Fetch-Reach evaluates reaching and obstacle avoidance rather than object manipulation. Moreover, the offline VLM dataset includes irreversible states, but the interactions and resets needed to collect these data are not reported or included in Fig. 4. Consequently, the total intervention cost and applicability to real-world tasks remain unclear.
  3. The comparisons do not sufficiently isolate the new contributions. Binary-search labeling comes from PAINT, and the state-information curriculum builds on Lee & Seo (2024) and Lee (2025). The baselines are not described as receiving the same curriculum, so Fig. 4 combines its benefits with those of the proposed estimator and transition policy. RSA (Lee, 2025), a closely related method for irreversible ARL, is also missing from the comparison. I would like to see controlled comparisons under the same curriculum, observation inputs, and oracle reversibility labels: a PAINT-style classifier, the proposed estimator without and with N-step expansion, and the addition of the transition policy. These should report both success rates and manual resets across all four environments. Demonstrating a clear benefit under these controls would substantially strengthen my assessment of the contribution.
  4. The ablations support component usefulness but do not establish the proposed design’s advantage. Fig. 6 compares the full method, a version without the transition policy, and a version without either component. This provides evidence that both components reduce resets, but only in two environments and without reporting success rates. It does not compare against a simpler estimator or quantify the downstream benefit of N-step expansion. Fig. 5 provides only visual evidence of earlier warnings, without lead-time or false-alarm measurements. Although the learning curves report means and standard deviations over five seeds, aggregate RL results and statistical comparisons are absent, and the observation inputs used by the baselines are insufficiently specified.
  5. The scope of the image-based claim needs clarification. The policies and curriculum use simulator state inputs (Sec. 3, Algorithm 1, Table 7); image observations are used for irreversibility detection. This limits the scope of the paper’s criticism of prior state-based methods. The authors should distinguish the detector’s observation requirements from those of the full system and explain which state inputs would be available in a real deployment.
  6. Important implementation details are missing. The manuscript does not specify N, τ_warning, τ_danger, or the penalty magnitude, and it does not define the reward incorporating that penalty. These choices directly affect the safety–performance trade-off and are necessary for reproduction. No implementation or code link is provided in the manuscript.
Questions for authors
  1. Supervision and total intervention cost. How many environment interactions and resets were needed to collect the offline VLM dataset? How would including this cost affect the reset comparison? Given the environment-specific fine-tuning, should the claim be narrowed to avoiding online human annotation?
  2. Isolating the contribution. Under the same curriculum, observation inputs, and oracle reversibility labels, how do a PAINT-style classifier and the proposed estimator—with and without N-step expansion—compare? What additional benefit does the transition policy provide? Please report success rates and manual resets, and compare against RSA (Lee, 2025). Evidence of a consistent benefit under these controls would substantially improve my assessment.
  3. Early-warning quality and implementation. What lead time and false-alarm rate does the estimator achieve, particularly in Highway-Parking? Please also specify N, τ_warning, τ_danger, the penalty and reward formulation, and the observation inputs used by each baseline.
4 comments

Thank you for carefully reading our paper and for your specific comments. In particular, your suggestions on the supervision requirements, the controlled comparison, and the quantitative evaluation helped us present our contributions more clearly. We have revised the paper to address all of your comments and marked all changes in blue.


[W1-1, W2-2 & Q1] Supervision requirements and cost of the offline VLM dataset

Your question on the supervision requirements raises an important point. It allowed us to describe the supervision requirements of our algorithm more precisely. We would like to note that our algorithm can work with a VLM without environment-specific fine-tuning, and the offline VLM dataset is therefore optional. The only reason we use a small fine-tuned VLM is labeling efficiency during ARL training. Thus, we believe that our claim does not need to be narrowed to avoiding online human annotation. Note that in our algorithm, the VLM is used only to provide a binary reversibility label for each image observation.

To verify this claim, we conducted additional experiments that compare our algorithm with the fine-tuned VLM and with a closed-source VLM without fine-tuning. We first compare the labeling performance of the two VLMs on the 2,000 held-out images. Without fine-tuning, GPT-6.1 Astra reaches an accuracy of at least 98.6% and an irreversible-state recall of at least 96.8% in all four environments, which is comparable to the fine-tuned Qwen3-VL (Table 8 of Appendix F.1).

EnvironmentMetricFine-tuned Qwen3-VLGPT-6.1 Astra
PointMaze-TrapAccuracy (%)99.1599.2
Recall (%)95.0297.6
AntOpen-TrapAccuracy (%)99.7098.8
Recall (%)99.7797.1
Fetch-Reach-TrapAccuracy (%)100.0099.4
Recall (%)100.0098.5
Highway-Parking-TrapAccuracy (%)98.7098.6
Recall (%)98.9896.8

Labeling performance on the 2,000 held-out images (Table 8 of Appendix F.1).

We then replace the fine-tuned VLM with GPT-6.1 Astra in our algorithm. This increases manual resets by no more than 4.4% in all four environments, with similar success rates (Table 10 of Appendix F.1). These results support our claim that the proposed algorithm does not need environment-specific fine-tuning.

EnvironmentMetricFine-tuned Qwen3-VLGPT-6.1 Astra
PointMaze-TrapSuccess rate (%)100.0100.0
Manual resets184186
AntOpen-TrapSuccess rate (%)91.790.0
Manual resets6871
Fetch-Reach-TrapSuccess rate (%)88.387.5
Manual resets467474
Highway-Parking-TrapSuccess rate (%)90.088.8
Manual resets837859

Success rates and manual resets of our algorithm with each VLM, mean over five random seeds (Table 10 of Appendix F.1).

In addition, we now report the cost of collecting the offline VLM dataset in Appendix B.2. This cost is incurred only when the fine-tuned VLM is used; with GPT-6.1 Astra, no offline VLM dataset is collected. The table below reports the time per query for labeling an image (Table 9 of Appendix F.1); GPT-6.1 Astra takes more than 37 times longer per query than the fine-tuned Qwen3-VL.

ModelTime per query (ms)
Qwen3-VL-2B, fine-tuned25
Qwen3-VL-4B, fine-tuned40
GPT-6.1 Astra, zero-shot1,500

Overall, our algorithm does not need any environment-specific supervision, such as fine-tuning data. Indeed, our additional experiments show comparable labeling accuracy and task performance even without a fine-tuned VLM. Tables 8, 9, and 10 are newly added in Appendix F.1.

[W1-2] Performance qualification (success rate & manual resets)

Your comment helped us state our results more accurately. In our work, what "outperform" means is achieving comparable success rates with much fewer manual resets. Manual resets are one of the key bottlenecks in autonomous RL, so we believe that requiring much fewer manual resets at comparable success rates is a meaningful advantage over the baselines.

We added a new table (Table 16 in Appendix F.6) to show that our algorithm requires the fewest manual resets in all four environments. As you mentioned, PAINT-MG and R3L-MG reach higher success rates in Highway-Parking-Trap (Table 16 of Appendix F.6). However, they require 2,048 and 6,510 manual resets, whereas our algorithm requires only 837. These results show that our algorithm requires the fewest manual resets in all four environments while achieving success rates comparable to those of the strongest baselines in three of them. Since each manual reset costs considerable time and money in the real world, this reduction in manual resets makes our algorithm more practical for real-world training. Please refer to Figure 4 in the main paper for the learning curves corresponding to the added table.

AlgorithmMetricPointMaze-TrapAntOpen-TrapFetch-Reach-TrapHighway-Parking-Trap
LNTSuccess rate (%)28.38.331.712.5
Manual resets2692131,069987
LNT-MGSuccess rate (%)46.735.063.350.0
Manual resets3643761,858951
PAINTSuccess rate (%)33.335.023.310.0
Manual resets2645165731,221
PAINT-MGSuccess rate (%)100.061.780.0100.0
Manual resets3724748742,048
R3LSuccess rate (%)33.353.345.012.5
Manual resets7801,3712,5394,733
R3L-MGSuccess rate (%)100.096.785.0100.0
Manual resets2,0192,1992,7266,510
RSASuccess rate (%)100.088.379.281.3
Manual resets2651421,1051,190
OursSuccess rate (%)100.091.788.390.0
Manual resets18468467837

Success rates (%) and manual resets, mean over five random seeds (Table 16 of Appendix F.6). The fewest manual resets are in bold.


[W2-1] Real-world motivation

Thank you for this thoughtful comment. We would like to note that our experiments reflect the real-world motivation. The irreversible regions are marked explicitly because such marked regions are common in the real world. For example, hazardous areas in factories and warehouses are marked on the floor with colored tape, parking slots that are unavailable are marked on the ground in a parking lot, and the workspace of a robot arm contains regions that the arm must not enter during a task. To reflect these situations, we use variants of existing environments that introduce irreversible states, namely the purple areas in PointMaze-Trap and AntOpen-Trap, the purple parking slots in Highway-Parking-Trap, and the wall-shaped area in Fetch-Reach-Trap.

In these environments, the only information that our algorithm uses about the irreversible regions is a noun phrase that names the visual cue. Unlike prior ARL algorithms, it requires neither a reset reward function nor human-provided reversibility labels (Appendix C), and it identifies these regions from image observations, which could be obtained from a camera in a real-world deployment. Thus, our experiments show that irreversible states can be identified from image observations without the privileged state information or human-provided reversibility labels that are difficult to obtain in the real world. We believe that this is an important step toward training ARL agents in the real world.

[W3, W4-1 & Q2] Isolating the contributions (controlled comparison)

The controlled comparison you suggested allowed us to separate our two contributions from the effect of the curriculum. Following your suggestion, we conducted the comparison between diverse conditions, which differ only in how they handle irreversible states, in all four environments (Table 12 of Appendix F.2). Condition (1) is a PAINT-style classifier, which predicts whether the current image observation is an irreversible state. Conditions (2) and (3) are our irreversibility estimator without NN-step expansion (N=0N=0) and with N=10N=10. Condition (4) is our algorithm, which adds the transition policy to (3), and condition (5) is RSA. Every condition uses the same curriculum with the state information estimator, the same policy inputs, and the same oracle reversibility labels from the simulator.

Manual resets continuously decrease from (1) to (4) in all four environments. Our irreversibility estimator requires at least 11.1% fewer manual resets than the PAINT-style classifier with N=0N=0 and at least 21.5% fewer with N=10N=10. Adding the transition policy reduces them by at least a further 11.8%. Our algorithm requires at least 28.8% fewer manual resets than RSA and achieves the highest or equal success rates in all four environments.

EnvironmentMetric(1) PAINT-style(2) Estimator (N=0N=0)(3) Estimator (N=10N=10)(4) Ours(5) RSA
PointMaze-TrapSuccess rate (%)100.0100.0100.0100.0100.0
Manual resets279248219181265
AntOpen-TrapSuccess rate (%)86.787.589.291.788.3
Manual resets1501319763142
Fetch-Reach-TrapSuccess rate (%)79.280.884.287.579.2
Manual resets1,1601,0108654551,105
Highway-Parking-TrapSuccess rate (%)83.883.882.585.081.3
Manual resets1,2401,0759608471,190

Success rates (%) and manual resets, mean over five random seeds (Table 12 of Appendix F.2). The fewest manual resets are in bold.

Since the curriculum is the same in every condition, these results show that the reduction in manual resets comes from the irreversibility estimator and the transition policy. Table 12 is newly added in Appendix F.2, and RSA is newly added as a baseline in Figure 4 and Table 16 of Appendix F.6.


[W4-2 & Q3-1] Early-warning quality

Your question led us to measure the early warnings quantitatively. To address your concern, we added a quantitative evaluation of the early warnings (Table 13 of Appendix F.3) and a sensitivity analysis of NN (Table 14 of Appendix F.4). We freeze the trained irreversibility estimator, collect rollouts without early aborts, and measure the lead time and the false-alarm rate. The lead time is the median number of steps from the first warning prediction to the irreversible transition. The false-alarm rate is the fraction of warning predictions that are not followed by an irreversible transition within NN steps.

MetricPointMaze-TrapAntOpen-TrapFetch-Reach-TrapHighway-Parking-Trap
Lead time (steps), N=0N=01111
Lead time (steps), N=10N=108879
False-alarm rate (%)9.411.87.623.5

Early-warning quality in the four environments (Table 13 of Appendix F.3).

NNLead time (steps)False-alarm rate (%)Manual resetsSuccess rate (%)
016.11,01283.8
5412.993185.4
10923.584785.0
201638.280276.3

Effect of NN in Highway-Parking-Trap (Table 14 of Appendix F.4).

NN-step expansion increases the lead time from 1 step to at least 7 steps in the four environments. In Highway-Parking-Trap, increasing NN from 0 to 10 reduces manual resets and raises the success rate, whereas increasing it further to 20 reduces manual resets only slightly and lowers the success rate. These results show that the irreversibility estimator issues warnings several steps before the irreversible transition. A larger NN issues warnings earlier and reduces manual resets, but it also increases false alarms and can lower the success rate. We therefore use N=10N=10. Tables 13 and 14 are newly added in Appendices F.3 and F.4, respectively.


[W6-1 & Q3-2] Implementation details

Your comment led us to add the settings needed for reproduction. To address your concern, we added the missing settings and the reward formulation (Table 2 of Appendix B.1). We use N=10N=10, τwarning=0.8\tau_{\mathrm{warning}}=0.8, τdanger=0.9\tau_{\mathrm{danger}}=0.9, and penalties of cwarning=−0.8c_{\mathrm{warning}}=-0.8 and cdanger=−0.9c_{\mathrm{danger}}=-0.9 in all four environments.

[W4-3 & Q3-3] Observation inputs used by the baselines

Your comment led us to specify the observation inputs used by the baselines. To address your concern, we added a table that lists the policy input of each algorithm separately from its input for identifying irreversible states (Table 11 of Appendix F.2). Every algorithm, including ours, uses state inputs for the policies. The algorithms differ only in how they identify irreversible states. LNT, PAINT, and RSA identify them from state inputs, and our algorithm identifies them from image observations. R3L does not identify irreversible states.

AlgorithmPolicy inputInput for identifying irreversible states
LNT, LNT-MGStateState
PAINT, PAINT-MGStateState
R3L, R3L-MGState–
RSAStateState
OursStateImage

Observation inputs of each algorithm (Table 11 of Appendix F.2).

Table 11 is newly added in Appendix F.2.


[W5] Scope of the image-based claim

Your comment helped us state the scope of our contribution precisely. We would like to note that learning policies from image observations was not the focus of this work, so, to keep ARL training fast, we used state inputs for the policies, such as joint angles and velocities. Learning policies from image observations has already been demonstrated in image-based reinforcement learning, such as DrQ-v2 [1], and in image-based ARL, such as MEDAL++ [2]. With image observations, we aim to show that irreversible states can be identified in ARL without privileged state information or a predefined reset reward function. In the real world, privileged state information requires additional instrumentation, and a predefined reset reward function requires domain knowledge of each task. Our algorithm requires neither.

We appreciate your constructive suggestions. The controlled comparison you suggested (Table 12 of Appendix F.2) shows a consistent benefit of the irreversibility estimator and the transition policy under the controls you specified in all four environments. If our response has addressed your concerns, we would greatly appreciate it if you could reconsider your score, and we are happy to discuss any further questions.


[1] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto, "Mastering visual continuous control: Improved data-augmented reinforcement learning," in International Conference on Learning Representations, 2022.

[2] A. Sharma, A. M. Ahmed, R. Ahmad, and C. Finn, "Self-improving robots: End-to-end autonomous visuomotor reinforcement learning," in Conference on Robot Learning, 2023.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their careful evaluations and constructive feedback. Based on your suggestions, we have added experiments and analyses that support our claims and clarified several aspects of the paper. We summarize the main revisions below.

Paper Revision Summary

All revisions are highlighted in blue in the revised paper.

  • We added experiments with a VLM without fine-tuning (GPT-6.1 Astra) in Appendix F.1 to show that our algorithm does not require fine-tuning a VLM. These experiments compare its labeling performance and time per query with those of the fine-tuned VLM and train our algorithm with it in all four environments. (Ge4M, 5eQL, 7Scr)
  • We added RSA to Table 1, Figure 4, and Appendix F.6, and compared it with our algorithm as a baseline in all four environments. (5eQL, 7Scr)
  • We revised the Abstract and Introduction to state more precisely that our algorithm achieves comparable success rates with much fewer manual resets. We also added a table in Appendix F.6 that reports the success rates and manual resets of all algorithms. (Ge4M)
  • We extended the ablation study to all four environments in Appendix F.2 to separate the contributions of the irreversibility estimator and the transition policy. (Ge4M, 5eQL, 7Scr)
  • We added a table in Appendix F.2 that lists the observation inputs of each algorithm, to clarify which inputs are used for the policies and for identifying irreversible states. (Ge4M)
  • We added a quantitative evaluation of early warnings in Appendix F.3 to examine whether the irreversibility estimator predicts entering irreversible states multiple steps in advance. In Appendix F.4, we also examined how the number of label expansion steps NN affects early warnings, manual resets, and success rates. (Ge4M, 5eQL, 7Scr)
  • We added statistics of the transition policy in Appendix F.5 and clarified how it is executed in Appendix A and Algorithm 2. (7Scr)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.