acceptodds
Withdrawn from ICLR 2027

Never Stop Rewiring: Safe Continual Reinforcement Learning on Non-Resetting Graphs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A network agent that is to remain autonomous over a long horizon must keep learning as failure processes arrive, vanish and return, and it must do so without ever breaking the network, because on a graph that is never reset a wrong repair rewrites the topology that every later failure acts on. Continual RL supplies the first requirement and safe RL the second, but each is studied without the other: continual RL on benchmarks that reset between tasks, safe RL in episodes that undo a violation. We argue that lifelong network repair needs their combination, and study it as task-free safe continual RL on a non-resetting graph: a non-stationary constrained MDP with a service reward, a connectivity safety cost, no mode labels, and a budget-B unrecoverable event, a violation that no B edge additions can repair. Across Erdos–Rényi, Watts–Strogatz and Barabási–Albert ˝ streams with recurring and unseen failure modes, we obtain three findings. Reward forgetting, the failure continual RL is built to prevent, is near zero for every learner; what ends autonomy is an absorbing violation state into which the service reward leads the agent and out of which later learning does not climb. Continual learning alone makes this worse: PPO is absorbed in 17 of 30 lifetimes, policy consolidation in 26, because the cascade stores the unsafe behaviour; a Lagrangian constraint alone only delays it (12 of 30). Their combination, constrained policy consolidation, which prices the constraint on the visible policy of a multi-timescale cascade, keeps rewiring for 29 of 30 lifetimes with the lowest violation rate in every graph family and a violation rate that falls on every return to a mode, without task boundaries. For a network to become autonomous over its whole lifetime, its agent must therefore never stop learning and never stop being constrained: the reward it can relearn, the safe behaviour it must carry forward.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual reinforcement learning, catastrophic forgetting, network reconfiguration, non-stationary environments

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jmdPOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper considers the problem of continual, safe reinforcement learning for rewiring networks (for example, infrastructure networks such as power grids). The argument behind the paper is that infrastructure networks experience different types of failures (e.g., failing components) at different stages and must keep functioning while keeping a reasonable cost of reconfiguring the network. While related works have studied constrained MDPs and continual RL in isolation, none have done so in the network rewiring setting. The paper loosely formulates the problem and proposes an RL approach for this setting. It is evaluated on synthetic networks with 50 nodes and mostly compares favourably with greedy approaches and vanilla RL, standalone safe RL, and standalone continuous RL.

Critical strengths

S1. The key idea behind the paper is interesting and novel, and identifies a reasonable challenge before reinforcement learning techniques for networks can be deployed to configure infrastructure networks in the real world.

Critical weaknesses

W1. Lack of clarity arising from LLM-driven writing

The key weakness of the work is its lack of clarity. The writing uses many specialist terms, concepts, and notations that are assumed to be known to the reader, but are often left undefined or assumed to be known throughout. To give examples:

  • The paper never explicitly mentions what is the action space, only mentioning rewiring in a loose sense; but there are many ways to frame rewiring mathematically.
  • The justification for combining the two RL techniques (end of Section 4) relies on continuous/safe RL notions that were never introduced.
  • Similarly, the metrics (5.1) are hard to follow since relevant background is missing.

These are just a few examples, but the lack of clarity is a pervasive feeling that affects every paragraph of the text. The writing is locally coherent, but lacking a clear narrative that can be followed sentence by sentence. The general feeling reading the manuscript is that it is LLM-generated to a very large extent, and not human-originated writing that was merely edited using LLMs.

At least the authors do acknowledge in the AI use statement that LLMs carried out writing under supervision. However, is this sufficient to justify authorship? Given the pervasive lack of clarity in the paper, are the authors qualified to judge whether the work is technically correct? These are timely questions that extend beyond this paper.

W2. Limited evaluation scope and magic numbers

The method is only evaluated on synthetic graphs with 50 nodes that are generated using "classic" graph generative models with fixed parameters (L272-281) and under very specific parameter settings for the environment (L294-L302). There are many "magic parameters" that are stated but whose choice is never justified. T

he coverage of the experiments should be expanded substantially in order to establish whether the results hold in a general sense, or whether they are specific to the very narrow slice of (graph instance, environment parameters) combination that was tested. There are no experiments on real-world graphs.

The transferability of policies to different graphs, particularly to those larger in size, would support the motivation for the method. Is the method applicable out-of-the-box to graphs of different sizes?

W3. Limited novelty

The method combines two well-known techniques and thus methodological novelty is very limited. The paper argues that the application is more novel, but in my opinion existing applications of RL to infrastructure networks are somewhat under-stated. The lack of clarity and factful-but-limited interpretation of results also lead to little insight being gained from the paper in my opinion.

Questions for authors

Please address W1, W2, and W3 as these are the key points that affect my assessment. I am also listing some further queries and suggestions for improving the work below.

C1. The rewiring action space should be explicitly defined in the paper (see also W1).

C2. Regarding Q3, I would suggest citing the following two existing works since they deal with topological reconfiguration of infrastructure networks under robustness and efficiency objectives. The citations on L145 are overly generic, making the work appear more novel than it is in my opinion.

[1] Darvariu, V. A., Hailes, S., & Musolesi, M. (2021). Goal-directed graph construction using reinforcement learning. Proceedings: Mathematical, Physical and Engineering Sciences, 477(2254), 1-18.

[2] Yang, S., Kaili, M. A., Wang, B., Yu, T., & Zha, H. (2023). Learning to boost resilience of complex networks via neural edge rewiring. Transactions on Machine Learning Research.

C3. In my opinion, the 2x2 comparison should be explained much sooner than L234, since it is referenced several times before that, including in the list of contributions.

C4. Why are only some metrics bolded in Table 1 but not others? What is the argument for not bolding W and F_R?

C5. Section 5.2 is devoid of interpretation and does not explicitly say what the conclusions regarding the research questions were. It is not necessary to restate results that are already visible in the figures, or redefine experimental protocol / definition of metrics. At least part of this section should be dedicated to drawing conclusions about the questions (currently left to the final section, too late). This is another hallmark of AI writing.

Reviewer_ZBPjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies safe continual RL without task boundaries on a non-resetting graph. An agent selects an absent edge to add to a 50-node network (ER, WS or BA), subject to a capacity ceiling. Hidden, recurring failure processes (random, hub, regional, and a late-arriving betweenness mode) delete bursts of edges. The reward measures service efficiency between demand pairs. The binary cost indicates whether the giant-component fraction or its algebraic connectivity relative to the initial graph falls below a threshold. The objective imposes a windowed average-cost budget of d = 0.05.

The authors evaluate a 2×2 factorial design over PPO, PPO-Lagrangian, PPO with policy consolidation (Kaplanis et al., 2019), and their combination, CPC. Lifetimes last 20,000 steps, with 10 seeds per graph family. They report three main findings:

  • Reward forgetting is close to zero for all learners.
  • Unconstrained learners are drawn by the service reward into an "absorbing violation state", defined as a violation run of at least 1000 steps that lasts to the end of the lifetime.
  • CPC is absorbed in 1 of 30 lifetimes, compared with 12 for PPO-Lagrangian and 17 for PPO, and has the lowest violation rate among the four learners in each graph family.

The paper also introduces a budget-B unrecoverability event and reports reward and safety forgetting separately. The appendices include budget, ordering and hyperparameter ablations and a boundary-aware Safe EWC reference.

Critical strengths
  1. The setting exposes an important gap between maintaining performance and maintaining safety. In a non-resetting graph, an action changes the state on which future failures act, so the consequences of learning persist across mode changes. Separating reward forgetting from safety forgetting is therefore useful, especially when reward forgetting is close to zero. The experimental protocol includes a separate tuning stream, matched graph seeds and mode schedules, and extensive ablations. The released logs also allow the main result to be checked: CPC has 1/30 terminal persistent-violation lifetimes versus 12/30 for PPO-Lagrangian, with all 11 discordant paired lifetimes favoring CPC (exact paired McNemar p ≈ 0.00098). Its advantage remains at a 200-step threshold, where the counts are 4/30 and 16/30. This is convincing evidence of improvement on the measured outcome.
Critical weaknesses
  1. The persistent-violation result is meaningful, but its long-term interpretation remains unresolved.

    A lifetime is classified as absorbed when its final violation run lasts at least 1000 steps and continues to T = 20,000. The released logs reproduce the reported counts, including CPC's 1/30. The issue is what this endpoint-dependent definition tells us about the underlying process. The authors explicitly avoid claiming global irreversibility and acknowledge that another policy can leave such states; the abstract and discussion should preserve those qualifications.

    The same trajectories illustrate why the observation horizon matters. At a cutoff of 15,000 steps, the counts are 4/30 for PPO-Lagrangian and 2/30 for CPC, compared with 12/30 and 1/30 at 20,000. Six of PPO-Lagrangian's final long runs begin after step 15,000, while a run classified as terminal at an earlier cutoff may subsequently end. Fig. 5b therefore describes the onset of runs that remain unresolved at the chosen endpoint, rather than an irreversible event recognizable when it begins. The results do not establish that constraints merely delay an inevitable failure.

    Long runs that eventually end are common. PPO-Lagrangian and CPC have at least one run of 1000 or more steps in 21/30 and 20/30 lifetimes, respectively, yet their observed fractions of time in such runs are 0.23 and 0.11. This is an important improvement, but it does not by itself identify faster recovery: onset, recurrence, mode exposure and right-censoring also affect occupancy. Reporting these components would clarify whether CPC mainly prevents, delays or shortens failures.

    Threshold sensitivity supports the qualitative comparison: at 200 steps, terminal-run counts become 16/30 for PPO-Lagrangian and 4/30 for CPC. The saved summaries agree with this threshold, whereas the current table script recomputes the 1000-step metric. The protocol history warrants clarification, without assuming that the threshold was chosen after inspecting outcomes. Overall, the evidence supports fewer terminal persistent violations and less observed time in long violation runs over the tested lifetime; stronger claims need duration analyses that account for censoring and checks over longer horizons.

  2. The safety objective and the specific benefit of the cascade need a clearer empirical interpretation.

    The stated budget is d = 0.05, but CPC's violation rates are 0.30, 0.30 and 0.50 on ER, WS and BA. These are improvements over PPO-Lagrangian's 0.35, 0.44 and 0.56, but they do not demonstrate budget satisfaction. Safety-greedy also exceeds the budget, at 0.145, 0.121 and 0.265. Failure of these methods does not prove infeasibility, and the short calibration stream without betweenness failures does not establish feasibility on the full evaluation streams. The distinction between reducing risk and meeting the stated constraint is central to interpreting the results.

    The service–risk trade-off deserves equal attention. Safety-greedy incurs approximately 41–53% of CPC's violations while giving up about 0.06–0.10 in service. The Lagrange multiplier also frequently reaches its cap in some BA runs, limiting further increases in the penalty. Neither observation invalidates CPC's improvement, but together they make it important to explain what level of safety the method is intended to achieve and at what cost in service.

    The factorial design and K/ω ablations support the tested combination of constraints and consolidation. They provide less direct evidence that the shadow policies retain safe repair skills, or that the cascade is preferable to other consolidation mechanisms. The Safe EWC reference changes both boundary access and the constraint formulation, so it does not isolate this comparison. A matched PPO-Lagrangian plus EWC baseline (Kirkpatrick et al., 2017; DOI 10.1073/pnas.1611835114), or direct safety probes of the shadow policies, would be informative. The already-planned PID multiplier (Stooke et al., 2020; arXiv:2007.03964) is another focused comparison, although it should not be expected to resolve infeasibility or saturation automatically.

    Finally, all results come from a single simulator with 50-node synthetic graphs, an add-only action space, an edge cap and one cost definition. These are reasonable choices for a controlled study, but may shape the persistent-violation behavior. The authors acknowledge several limitations; narrowing the broad conclusion about lifelong network autonomy would bring the claims into closer alignment with the evidence. Further comparisons should assess both service and violations under matched tuning and information access.

Questions for authors
  1. Metric definition and protocol history. Could you clarify when the 1000-step threshold was selected relative to the saved summaries that match a 200-step threshold? It would also help to explain Fig. 5b as a curve of terminal-run onsets conditional on the observation horizon, and indicate how its interpretation changes under alternative thresholds or endpoints.

  2. Prevention, delay or recovery. Can the released logs distinguish how CPC reduces persistent violations? CPC and PPO-Lagrangian have similar numbers of lifetimes containing a long violation run, but different occupancy in those runs. Analyses of onset, recurrence and duration, accounting for terminal censoring and mode exposure, would help separate the explanations; a longer-horizon check would provide additional evidence.

  3. Meaning of the safety objective. What evidence supports feasibility of d = 0.05 on the full evaluation streams, given that even the safety-greedy baseline exceeds it? Could you clarify how the observed violation rates, window-level budget exceedances and multiplier saturation should be interpreted relative to the stated objective? Distinguishing adaptation-related violations from persistent or externally induced ones would also help.

  4. Evidence for retained safety knowledge. Do the shadow policies themselves retain safer behavior when earlier modes return? Direct probes would help connect the cascade to the proposed storage of safe repair skills. Alternatively, a matched PPO-Lagrangian plus another consolidation method, with clearly specified boundary access, could help assess whether the benefit is specific to the cascade rather than consolidation more generally.

  5. Service–risk trade-off and controller choice. How does CPC compare with a stronger constraint controller, such as the already-planned PID multiplier, when both service and violation rate are considered? A comparison under matched tuning effort would help assess the contribution of the cascade. It would also be useful to discuss where CPC is preferable to safety-greedy, which achieves fewer violations at some cost in service.

Reviewer_ChRaOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates safe continual RL on non-resetting graphs. The paper proposes a constrained policy consolidation method that combines PPO-Lagrangian with policy consolidation to restore network connectivity under changing failure modes.

Critical strengths
  1. The paper studies an interesting problem.
  2. The paper combines safe RL and continual learning.
Critical weaknesses
  1. The paper is in general hard to understand. Even for technical details, the paper does not present them in a clear and straightforward manner, but instead often uses overly elaborate expressions and convoluted sentence structures. The paper reads more like literary prose than academic writing. I suggest the authors use more direct and precise language to improve readability.

  2. Many mathematical notations are not clearly defined the first time they appear. For example, rtr_t and ctc_t appear in Section 3.1 but are not defined until Section 3.2. dGd_G does not seem to be defined throughout the paper. It is not clear whether WW in Eq.(1) is the same as WW in Eq. (3). Below Eq.(6), it is not clear what LV\mathcal{L}_V and H\mathcal{H} are. Overall, the lack of clear definitions and rigorous mathematical presentation makes it difficult to understand the paper's technical details.

  3. The paper seems to handle the constraints using a Lagrangian method, which incorporates a soft constraint rather than providing any formal safety guarantees. There is no proof that the constraints can be formally satisfied.

  4. The experiment looks rather weak. First, the three scenarios are idealistic networks with limited practical significance. Second, the experiments seem to lack diversity. All scenarios share similar settings, e.g., 8 demand pairs, same strike probalities and edge removal process, etc. In other words, these experiments look more like sensitivity analyses on the network topologies. Third, please include more state-of-the-art methods and present them in the main text.

  5. It is not clear why continual learning is needed. It seems from Table 1 and Figure 4 that FRF_R and FCF_C are both close to zero, meaning that even the standard PPO may not have significant catastrophic forgetting. It is not clear whether statistically, FRF_R and FCF_C are significantly different from zero. In general, the statistical analysis of reward and safety forgetting seems insufficient.

Questions for authors
  1. How is safety formally guaranteed?

  2. Why is continual learning needed in your method?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.