acceptodds
Under review as a conference paper at ICLR 2027

The Fate of a Failed Training Run: Recovery Dynamics and Replay-Buffer Pathology in Soft Actor-Critic

Abstract

A failed reinforcement-learning run is usually reduced to one observation: its final score. We instead save the failed state and branch it repeatedly, turning post-failure recovery into a distribution over stochastic continuations. The resulting 720-run study covers 20 failed Soft Actor–Critic checkpoint branch points on two Meta-World tasks. Recovery is a multi-stage process: a first successful episode (a spark), optional crossing of a repeated-success marker (consolidation), and a final solved or unsolved evaluation; a consolidated failure is a relapse. No unsparked continuation solves (0/313), whereas 111/159 continuations that reach 50 successes solve. Critic-loss excursions are not terminal: 94/120 solved continuations cross the operational instability marker, and in 49/90 solved continuations containing both events the critic crosses first. Actor–critic reset raises the equal-checkpoint consolidation rate by 0.160 (95% checkpoint-blocked bootstrap interval ), although its solve-rate gain is less certain and heterogeneous across checkpoints. Replay data supplies a sharper intervention. On checkpoint s18, reset with the failed run's buffer yields 0/10 solves; clearing that buffer yields 4/10; and replacing it with two compatible zero-success foreign buffers yields 9/10 and 7/10. The matched donor comparison shows that successful donor episodes and post-reset donor history are not necessary for rescue in this setting, while leaving the enabling property of the replacement data open. Finally, a checkpoint-aware stopping frontier quantifies the cost of acting on delayed recovery: an illustrative k no-spark rule saves 24.6% of post-branch compute while sacrificing 7/120 eventual solves. Replicated branches therefore expose both the dynamics of recovery and a data-state obstruction that a parameter reset alone can preserve.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, Soft Actor-Critic, replay buffers, training instability, checkpoint recovery, plasticity, robotic manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_U42qOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates failed training runs in MetaWorld. There's no clear vision of the paper's main claim or even of the research presented here. Basically AI-slop

Critical strengths

None

Critical weaknesses
  • This paper is clearly AI-generated, including text, figures, and most likely most of the experiments.
  • The paper does an extremely poor job of presenting its claims and evidence. Basically, it is unreadable
  • The evaluation setup is just two environments.
Questions for authors

Please do not rely on LLMs to such an extent.

Reviewer_xKfqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the recovery of training dynamics from a failed checkpoint and launches multiple independent continuations under different interventions. The authors define spark, consolidation, solve, relapse, and critic excursion to describe post-failure trajectories. Using SAC on two robotic manipulation tasks (pick-place and push), they build a frozen, audited event table of 720 completed continuations across 20 failed SAC checkpoint branch points. They find that recovery is multi-stage, and that replay interventions such as clearing or replacing the buffer can rescue a failed checkpoint.

Critical strengths
  1. This paper studies an interesting but often ignored problem.

  2. The finding that the replay buffer can retain the failure is important.

Critical weaknesses
  1. The writing of this paper seems to be driven by LLMs. The abstract contains too many figures. Currently, it is not friendly for humans to read.

  2. Although the finding is interesting, its novelty is not sufficient, and its contribution seems thin. Maybe further utilizing this recovery mechanism to develop a more robust and powerful RL algorithm would strengthen its contribution.

  3. Whether the recovery could be applied to other tasks and algorithms is unknown.

Questions for authors
  1. Could the authors tell us when a failed checkpoint can be recovered?

  2. Why is clearing and replacing the replay buffer useful? Maybe parallel environment sampling could also avoid the failure. Do the authors consider this intervention?

  3. Why is actor-critic reset inconsistent in solving the failure? How about the soft reset, which partially resets the network parameters?

Reviewer_NM7dOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper replicates a "failed" SAC checkpoint (evaluation reward ≤ 100) into multiple independent training runs, running a total of 720 training runs across 20 branches in two Meta-World tasks (pick-place, push). The experiment is clearly insufficient.

Critical strengths

The branch checkpoint design has some value, treating a failed run as a distribution of training results and performing block-based statistical analysis by checkpoint. However, this advantage is insufficient to compensate for the following problems.

Critical weaknesses
  1. An experiment labeled "rebuttal" appeared directly in the initial submission (Table 2 title, Appendix F).

  2. The buffer size was misrepresented as the number of transitions (49,999 is approximately 200k), resulting in an exaggerated volume comparison of about 4 times.

  3. Consolidation is defined as ≥ 50 successful runs, but the text refers to a run with only 41 successful runs as "consolidated relapse."

  4. A paper discussing "recovery dynamics" lacks any training curves. Fig. 2a shows 15 overlapping lines that are difficult to read. Fig. 3 compares groups with different configurations, but page 7 has a large blank space. Overall, it does not meet top conference standards.

Questions for authors
  1. Why do experiments labeled "rebuttal" appear in the initial submission?

  2. After switching to the official Meta-World success metric and using different consolidation thresholds, do the main conclusions still hold true?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.