acceptodds
Under review as a conference paper at ICLR 2027

Recurrent Structure Reveals How Goal-Conditioned Policies Fail

Abstract

Failed trajectories may follow different paths yet converge to the same persistent outcome. We propose a behavioural interpretation of goal-conditioned policy failure whose unit is a recurrent outcome of the frozen policy–environment dynamics, its incoming basin, and its operating probability mass. This identifies which starts share a failure, where behaviour becomes trapped, and how consequential that failure is. In finite deterministic systems, the closed-loop map yields an absorbing goal outcome or non-goal fixed points and cycles. Across FourRooms and MiniGrid KeyDoor, this representation reveals failure concentration and task-stage bottlenecks. In the larger 16 × 16 KeyDoor setting (MG16), exact analysis identifies 1,755 goal-conditioned failure structures across 84 right-room goals. Discovery from 32 starts per goal—8.2% of the evaluated start population—recovers 85% of pooled recurrent failure mass and identifies the dominant failure in 98.8% of goal–trial pairs. With 64 starts, mean per-goal coverage reaches 91.3%. Discovery uses sampled executions; exact enumeration supplies evaluation ground truth. The representation also guides intervention. Sample-selected FourRooms edits redirect incoming basins toward success, while iterative MiniGrid repair addresses downstream failures. Recurrence-guided exploration and replay improve terminal right-goal success over ordinary continuation in all three tested MG16 seeds, with heterogeneous gains. These results support the recurrent outcome, basin, and mass as a recoverable and actionable unit of behavioural interpretation.

Primary area:
interpretability and explainable AI
Keywords:
Reinforcement Learning, RL Interpretability, Policy Interpretability, Behavioral Interpretability, Policy Dynamics, Recurrent Structures, Failure Modes, Policy Repair, Goal-Conditioned Reinforcement Learning, Closed-Loop Dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_Nn5LOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce an interpretability framework designed to analyze the persistent failure behaviors of goal-conditioned RL policies. The core motivation is that standard metrics—such as success rate and expected return—only quantify how poorly a policy performs, but fail to reveal how these failures are structurally organized (e.g., whether a high failure rate stems from a single pervasive mode or numerous isolated traps). To address this, the method models the closed-loop interaction between a frozen policy and the environment as a functional graph over a finite, deterministic state space. Since every state has exactly one successor, any trajectory must inevitably converge either to the goal (a successful absorbing state) or to a non-goal fixed point or cycle. The authors thus define the fundamental unit of failure interpretation through three components: the recurrence failure structure itself, its basin of attraction, and the probability mass absorbed by this basin under the operational distribution. Empirically, the framework is validated across three environments (a 12×12 FourRooms, an 11×9 MiniGrid KeyDoor, and a larger 16×16 KeyDoor). The results demonstrate that sampling merely 32 initial states is sufficient to recover an average of 85% of the recurrent failure mass and identify the primary failure structure in 98.8% of cases. Crucially, these discovered structures can effectively guide localized policy repairs—significantly boosting success rates with minimal edits—and direct additional exploration and replay to outperform vanilla continued training. However, the framework strictly relies on the assumptions of finite and deterministic state spaces, precluding direct application to continuous, stochastic, or partially observable settings. Furthermore, there is notable heterogeneity in the continuation experiments (e.g., the random-trigger baseline occasionally outperforms the guided approach on primary metrics), and the current experimental protocol conflates recurrence localization with local exploration and guided replay, making it impossible to attribute the observed performance gains strictly to the localization component itself.

Critical strengths
  1. Rigorous formalization of core concepts. Modelling the closed-loop interaction between a frozen policy and the environment as a functional graph—and systematically deriving the three quantities (recurrence structures, basins of attraction, and basin mass)—equips this work with a precise vocabulary that most interpretability studies lack. This triad design is compelling because it simultaneously answers three fundamental questions: what the failure behavior is, which starting states lead to it, and how severe it is. In contrast to success rates that collapse these dimensions into a single opaque scalar, this formulation preserves critical diagnostic granularity.

  2. A refreshing level of experimental transparency well above the ICLR average. The authors proactively disclose a series of methodological limitations and caveats: seed 0 in MG16 was used during protocol development (a developmental seed); the "Medium" and "Good" checkpoints in MG11 stem from the same training run rather than independent runs; the continuation experiments cannot cleanly decouple recurrence localization from exploration and replay; and certain log data were lost and unrecoverable. Such proactive candor is rare in submitted manuscripts and effectively mitigates the risk of reviewers or readers overinterpreting the claims.

  3. Concrete and statistically backed sampling recovery findings. Instead of settling for qualitative claims like "sampling can recover most failures," the paper provides precise statistics: in MG16, sampling merely 32 initial states per goal (8.2% of the evaluation population) recovers an average of 85% of the recurrent failure mass and identifies the primary failure structure in 98.8% of cases. More importantly, the authors do not merely report the numbers; they explain why such low-sample recovery works—namely, that failure mass is heavily concentrated, with the top 5 structures accounting for an average of 78.7% of a goal's failure mass.

  4. Mechanistic and convincing intervention experiments. The patching experiment in FR12 is particularly persuasive: a single-step policy edit broke a self-loop, successfully redirecting all 45 initial states within that basin toward success, which boosted the goal's success rate from 44.4% to 80.6%. This goes beyond merely reporting performance metrics; it directly demonstrates the predictive utility of the "basin" concept regarding the scope of an edit's impact. Furthermore, the iterative repairs in MG16 illustrate that the framework correctly anticipates how breaking one failure mode will expose downstream failures, neatly justifying the architectural design choice of re-diagnosing every round.

  5. Task-phase decomposition uncovers insights masked by aggregate metrics. Categorizing recurrent failures by task phases (e.g., before acquiring the key, post-acquisition pre-opening, and inside the right room) reveals insights invisible to aggregate statistics. The paper's running example is striking: while 64.7% of the failure mass occurs before getting the key at the population level, for a specific target like (9,11), 60.6% of the failure mass occurs inside the right room. These orthogonal diagnostics point to entirely different intervention strategies, proving that goal-conditioned decomposition delivers genuine, actionable information.

Critical weaknesses
  1. Severe limitations in scope due to restrictive assumptions, with no clear path for generalization. The framework relies heavily on three stringent conditions: a finite state space, deterministic policies and environments, and precise state-revisitation detection. None of these hold in mainstream RL domains involving continuous control, stochasticity, or partial observability. While the authors acknowledge this in the Discussion, they offer virtually no viable roadmap for generalization. Consequently, all empirical conclusions are locked within a narrow grid-world domain, leaving readers to question whether the framework holds any practical reference value for broader, more realistic scenarios.

  2. Insufficient statistical backing for the core claims of the MG16 continuation experiments. Table 4 serves as the primary empirical evidence supporting the claim that the framework effectively guides interventions. However, in seed 2, the random-trigger baseline actually outperforms the guided method on the primary metric of right-goal success (63.7% vs. 48.5%), an anomaly brushed aside as "heterogeneous gains" without a proper explanation. Furthermore, the sample size of three seeds is inherently limited, and seed 0 was used during development rather than serving as an independent confirmatory run. More fundamentally, the experimental protocol tightly binds three distinct mechanisms—recurrence localization, local exploration, and guided replay—making it impossible to decouple their individual contributions. As the authors themselves admit, this leaves the core assertion that "recurrence localization is useful" effectively unvalidated in the continuation experiments.

  3. The distinction from AEGIS-RL remains purely textual. AEGIS-RL similarly constructs behavioral explanation graphs from sampled trajectories and connects them to policy interventions, a similarity the authors correctly note twice in the Related Work as their "most relevant precedent." While the conceptual distinctions drawn in the paper are valid, the absence of quantitative comparisons on any shared environment leaves a critical gap. Without head-to-head metrics, readers cannot determine whether the two frameworks yield divergent diagnostic or intervention outcomes, making it difficult to gauge this work's true incremental contribution over its closest prior art.

Questions for authors

Q1. In Seed 2, the "random-trigger" approach outperforms "recurrence-guided" in terms of "right-goal success" (63.7% vs. 48.5%)—a result that runs counter to the paper’s primary performance metric. The exposure audit in Table 21 reveals that, for Seed 2, the guided arm involved fewer actual trigger occurrences and fewer retained "targeted transitions" compared to the random arm. Do the authors believe this disparity in exposure is sufficient to explain the reversal observed in Seed 2? If not, what post-hoc hypotheses do the authors have regarding this result?

Q2. AEGIS-RL also constructs behavioral explanation graphs from sampled trajectories and links them to interventions for failure cases; the authors characterize it as the "most relevant precedent," distinguishing it from their own work based on the examination of temporal specifications and the decomposition of failure quality. However, this distinction appears to be at the level of objectives rather than methodology. In a shared environment, would these two frameworks identify different failure structures or yield different intervention recommendations? If the authors believe the outcomes would indeed differ, could they provide at least one concrete example illustrating the source of this divergence?

If my issue can be resolved in a reasonably satisfactory manner, I will consider increasing the score.

2 comments

We thank the reviewer for the constructive assessment and for recognizing the value of the structural diagnosis and the paper's experimental transparency.

Q1. Can unequal exposure explain the seed-2 reversal?

Unequal realized exposure is a genuine confound, but available evidence does not establish it as a sufficient explanation. All arms continued for 250k steps from the same seed-specific checkpoint. Guided and random used comparable nominal trigger-selection budgets, but actual encounters differed.

Our additional audit of the original code and saved records finds trigger activations (guided vs. random) of 1,910 vs. 2,363 (seed 0), 1,942 vs. 3,095 (seed 1), and 2,365 vs. 3,473 (seed 2). In seed 2, guided also retained fewer targeted transitions (48,185 vs. 71,589); these are buffer additions, not measured replay draws.

Random thus triggered more often in all three seeds, although guided had higher terminal right-goal success in seeds 0 and 1 (19.4% vs. 8.8%; 81.7% vs. 3.6%). Random won in seed 2 (63.7% vs. 48.5%). Higher exposure therefore does not consistently predict the ranking, but may help explain seed 2.

One hypothesis is that random triggers occur more often along frequently traversed paths, whereas guided triggers concentrate in or just upstream of recurrent failures. Random locations are selected uniformly from distinct diagnostic-trajectory states, excluding the recurrent neighbourhood, rather than weighted by training visitation. However, historical state-level visitation and trigger-location records were not retained. We cannot test this hypothesis or quantify its contribution to seed 2.

Accordingly, continuation is exploratory evidence about the combined localization, exploration, and replay procedure, not an isolated causal estimate of localization. More independent seeds, confidence intervals, and controls matched on realized exposure are needed.

Q2. What is the methodological distinction from AEGIS-RL?

We agree that AEGIS-RL is a close precedent: both derive behavioural graphs from trajectories and relate them to intervention. The difference is not merely the intended application, but how the graph is constructed and what its nodes represent.

AEGIS-RL constructs a semantic abstraction by clustering concrete states, then estimating transitions among abstract nodes for explanation and temporal-property verification. Our method does not start from semantic clustering. For a frozen deterministic policy, it groups starts by their exact eventual recurrent outcome—a particular non-goal fixed point or cycle—and measures the mass of its incoming basin. Limited executions discover these structures without exhaustive graph construction.

A concrete example is our FR12 repair: 45 different starts reach one non-goal fixed point. We identify their shared outcome and show that changing one action at the fixed point redirects the basin, raising success from 55/124 to 100/124. A semantic abstraction could merge states with different eventual outcomes or separate states that share one outcome; its nodes need not preserve that exact basin. Consequently, it could highlight different diagnostic or intervention locations.

This is an illustration, not a measured head-to-head result. AEGIS-RL might recover overlapping failures with appropriate predicates and specifications. Comparing both on common policies, matched diagnostic budgets, and measures of failure coverage and intervention utility would be necessary to establish their relative strengths.

Scope and generalization

We acknowledge the restriction to finite deterministic systems with exact state-repetition detection. Stochastic dynamics suggest replacing deterministic basin membership with absorption probabilities into recurrent classes; continuous or partially observable settings would require approximate recurrence in suitable state representations. Neither extension has been validated here.

In summary, the core contribution is sample-efficient recovery of consequential recurrent failures, with repair demonstrating potential actionability. The three-seed continuation study does not establish statistically significant superiority or isolate recurrence localization's contribution.

Thanks for the updates. I'll keep my score.

Reviewer_piyNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper aims to analyze the failure mode of goal-conditioned policies. They then trace out the trajectory taken by the policy from an initial state to a goal state and study when the agent ends up looping in a set, or a set of states and never leaves: something that the authors call recurrent failures. They then measure empirically the fraction of times that they end up in those recurrent loops, starting from an initial state. They carry out experiments in small grid worlds, four rooms, and in two mini-grid key-and-door tasks. Then show that if they start from about a dozen start states per goal, that's enough to capture most of the failure modes. Once they've identified the loops, they then propose to repair the policy by actually intervening and changing the action taken at that looping state. Or they could also decide to explore what to replay as they keep on training. They show that this repair strategy helps improve the final success on only three seeds.

Critical strengths

The paper is relatively careful and clear about the limitation of their experiments and the fact, for example, that the two checkpoints that they use from one run are not quite independent seeds, and that they only use one seed to develop their methodology. Their choice of the toy environment four rooms is quite adequate to make the point they want to make, but of course it's also very limited. See critical weakness in the following box.

Critical weaknesses

Theoretically, this paper really hinges on the standard decomposition of Markov chains into their transient states and recurrent classes. This is somehow acknowledged by the authors (the "graph" connection), but it is a fact that is underutilized throughout the paper. The paper makes a specific assumption that the system they study, the policy, and the environment itself are deterministic, which simplifies their approach and their enumeration argument. However, had they embraced the Markov chain perspective and standard decomposition tools, they could have also tackled the stochastic counterpart to their problem with readily established tools. For example, the absorption probability simply becomes a problem of solving a linear system. If you have a closed-loop chain and you want to know the probability of ending in any recurrent class for any given state, all you got to do is go through the limiting matrix (which is pretty much what is being rediscovered in the paper's "basin mass"). If you want to go further, I would even recommend reading Puterman's textbook. In the appendix section, you're going to find a very interesting section about the decomposition of MDPs in the discounted setting using the Laurent series expansion. This tool gives two parts:

  1. One that pertains to the transient part along the trajectory to a goal
  2. The other part pertaining to which recurrent class you end up in, whether it's a goal or a failure

Another important weakness of the paper is their continued training setup, where they only used three seeds and one of those seeds was also used to build the experimental protocol. Furthermore when they evaluate the control where they trigger some extra exploration at random places, they don't allocate the same amount of extra experience and that amount varies across seed. Which means here that it's difficult, in fact, to disambiguate if there is truly an effect.

Questions for authors

Why did you choose to build your own terminology and framework rather than relying on the existing theory of marker chains? Are there things that your framework allows you to say more naturally than the standard tools would otherwise be capable of doing?

How do you envision applying your repair changes if you were to be learning in a continuous control, or let's say a pixel-based policy, where a single state wouldn't be something you could override easily?

And most importantly, would you be able to provide enough seeds in the continued training setup so that you could compute a confidence interval on the benefits of your proposed method in addition to giving the guided and the random version as much extra experience as you provide for your method ?

1 comment
AuthorsOpenReview ↗

Markov-Chain Foundations, Sample-Based Discovery, and Experimental Scope

We thank the reviewer for recognizing the experimental limitations and for drawing attention to classical Markov-chain theory.

Q1. Why not use Markov-chain theory?

We agree that transient states, recurrent classes, and absorption probabilities are established mathematics; we claim no novelty for that decomposition. Our research question is whether consequential recurrent failures of learned goal-conditioned policies can be identified from a limited number of executions, without constructing the full closed-loop transition system.

With a complete transition model, classical methods identify recurrent classes and absorption probabilities. Our emphasis is different: sample-efficient behavioural diagnosis when building and analysing that model for many goals is costly. We measure the share of failure mass attributable to structures discovered through sampling, rather than simply counting structures. With 32 sampled starts per goal, recovery reaches 95.9% mean recurrent failure mass in FR12 (five policies) and 85.0% pooled mass in MG16. Exact censuses provide evaluation ground truth; we do not claim general scalability.

The reviewer is right that stochastic basin membership generalizes to absorption probabilities obtainable through linear systems. Puterman's discounted-MDP decomposition also offers relevant foundations. A future question is how to discover consequential probabilistic failure structures from trajectories without a complete model. We do not claim to solve that problem here.

Q2. What about continuous control or pixel observations?

Exact-state action overrides, as demonstrated in FR12, do not directly transfer. The potentially transferable principle is localizing interventions using recurrent failure structure: identifying approximately recurrent regions in a learned representation, then targeting local policy updates, exploration, or replay. MG16 continuation tests structure-guided exploration and replay without prescribing replacement actions. Reliable identification and repair of recurrent regions in continuous or pixel-based environments remain untested.

Q3. More seeds, confidence intervals, and matched exposure?

We agree that three continuation seeds cannot establish statistically significant superiority. Seed 0 was developmental; seeds 1–2 followed the fixed protocol. Additional independent seeds and confidence intervals are needed.

Our additional code-and-record audit confirms similar nominal trigger-selection budgets and fixed 250k-step continuations, but unequal realized trigger activations:

SeedGuidedRandom
01,9102,363
11,9423,095
22,3653,473

Random activated more often in every seed, yet guided achieved higher terminal right-goal success in seeds 0 and 1; random was higher in seed 2. Exposure therefore does not consistently predict the ranking, but it remains a confound. The continuation results concern the combined localization, exploration, and replay protocol, not an isolated causal effect of localization. A decisive comparison requires more seeds and matched realized exposure, not only matched nominal budgets.

In summary, this paper investigates sample-efficient recovery of consequential recurrent failures and their utility for intervention. It does not claim new Markov mathematics, general transfer to high-dimensional settings, or statistically proven training superiority.

Reviewer_MNgoOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a framework for understanding failures of goal-conditioned policies by identifying invariant sets (fixed points and periodic orbits) of the induced closed-loop dynamics, characterizing their domains of attraction, and estimating the associated probability mass under a distribution of initial conditions. They evaluate the recovery of these structures from sampled trajectories and investigate their use for policy diagnosis and intervention in deterministic grid-world navigation environments, including FourRooms and MiniGrid KeyDoor.

Critical strengths

None

Critical weaknesses
  • The paper does not clearly articulate the problem it is trying to solve or why the proposed framework is necessary. In Section 3, numerous mathematical objects are introduced without first establishing a concrete research question that motivates them. For example, the paper defines "recovered recurrent failure mass," but it is unclear why recovering this quantity is important or what practical problem it addresses. Is the primary goal to improve interpretability, identify failure modes, or guide intervention? These objectives are related but distinct, and the paper does not clearly establish what constitutes success for each.
  • The paper does not convincingly establish how the proposed framework differs from or improves upon existing methods for policy interpretation, failure analysis, and targeted intervention. The concepts introduced in the paper (e.g., representing the closed-loop dynamics as a graph, identifying invariant sets, finding their domain of attraction) are not new. The also experiments lack direct comparisons with existing methods. Consequently, it is difficult to assess whether existing approaches could already address the same problems, or whether the proposed method provides meaningful advantages.
Questions for authors
  • What is the central research question, and what practical problem does the proposed framework solve?
  • What advantages does the proposed framework offer over existing approaches?
2 comments

Response to Reviewer MNgo

We thank the reviewer for their detailed comments. We recognize that the paper could communicate its central research question and the relationship between its mathematical formulation, empirical analysis, and interventions more clearly. We address the two questions directly.

Q1. What is the central research question, and what practical problem does the framework solve?

Our central research question is:

Can we identify, characterize, and recover consequential persistent failures of learned goal-conditioned policies from limited policy executions, without enumerating the complete state space?

Aggregate success rates reveal how frequently a policy fails, but not how those failures are organized. For example, many unsuccessful starting conditions may converge to the same persistent behavioural trap, while others may lead to entirely different failures. Distinguishing these cases is useful both for explaining policy behaviour and for identifying consequential intervention targets.

Our primary objective is behavioural interpretability through recurrent structure. We investigate three connected questions, each with a distinct evaluation criterion:

  1. Structural diagnosis: Can we identify recurrent failure outcomes, determine which starting conditions converge to them, and quantify their importance through incoming-basin mass?
  2. Sample-based recovery: Can limited trajectories discover the recurrent structures accounting for most failure mass, without exhaustive enumeration? This motivates recovered recurrent failure mass: the fraction of the true recurrent failure mass associated with structures discovered through sampling.
  3. Actionability: Can identified recurrent structures provide useful intervention targets, measured through changes in policy success following repair or continued learning?

These stages serve a common purpose: moving from knowing how often a policy fails to understanding where persistent failure occurs, which starts share that failure, and where intervention may be valuable.

The empirical results illustrate this progression. In FR12, sampling 32 starts per goal recovers 95.9% of mean recurrent failure mass across five seeds. In a separate repair demonstration, a single policy edit at an identified recurrent fixed point redirects its 45-state incoming basin to successful flow, increasing successful starts from 55/124 to 100/124.

Thus, recovered failure mass measures whether limited sampling identifies the failures that matter most under the specified operating distribution, rather than merely counting how many distinct failures are discovered.

Q2. What advantages does the framework offer over existing approaches?

Our framework offers a specialized approach to behavioural diagnosis: it identifies shared persistent failure outcomes, quantifies their importance through incoming-basin mass, and investigates whether consequential recurrent structures can be recovered from limited policy executions without exhaustive graph construction. Our experiments demonstrate these capabilities in finite deterministic environments and show that the discovered structures can guide intervention. Establishing their relative advantages over existing policy-interpretation and failure-detection methods requires a carefully designed comparative protocol using common policies, sampling budgets, and metrics for diagnostic coverage, sample efficiency, and intervention utility. We view this as an important next step.

My concerns have not been addressed. It does not seem like the authors have modified the paper in any way.

Reviewer_5WqdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Despite spending an hour reading the abstract and introduction multiple times, I don't understand what problem the paper is trying to solve. My best guess is that it wants to say most failing trajectories in goal-conditioned RL originate from a small fraction of starting states, or get stuck in a subset of states that the paper calls as a basin or a recurrent outcome. I don't know why that is important, but it seems like the paper is then trying to change something in the training loop to turn these failing trajectories into successes.

Critical strengths

None.

Critical weaknesses

The abstract is very difficult to understand and tells the reader nothing about what problem the paper is trying to describe. It is full of undefined jargon. Some examples: "whose unit is a recurrent outcome of the frozen policy–environment dynamics, its incoming basin, and its operating probability mass." all undefined jargon. It is also filled with meaningless metrics that doesn't help the reader: "exact analysis identifies 1,755 goal-conditioned failure structures across 84 right-room goals. Discovery from 32 starts per goal—8.2% of the evaluated start population—recovers 85% of pooled recurrent failure mass and identifies the dominant failure in 98.8% of goal–trial pairs."

The introduction is worse, having 0 citations and thus placing this paper nowhere amongst related works, if any. Instead, it is riddled with vacuous statements such as "The graph mathematics is established; our contribution is this interpretive construction, its recovery from sampled executions, and its use for diagnosis and intervention."

The empirical protocol is unsound and not statistically significant. "Seed 0 is developmental; seeds 1–2 are confirmatory." Instead, the reader is bombarded with meaningless statements talking about metrics that are not interpretable "Across five FR12 policies, 125 goals per policy, and 124 non-goal starts per goal, the exact census covers 77,500 conditions. Success accounts for 57.12% of operating mass; 39.26% enters non-goal fixed points and 3.62% enters multi-state cycles. These failures comprise 2,376 goal-conditioned recurrent structures."

Questions for authors

What are the authors trying to do here?

2 comments
AuthorsOpenReview ↗

Clarifying the Research Objective, Sample-Based Failure Discovery, and Experimental Scope

We thank the reviewer for the detailed feedback. We agree that the abstract and introduction should explain the research problem and the meaning of the metrics before introducing technical terminology.

1. What are we trying to do?

Our central question is whether consequential recurrent failures in learned goal-conditioned policies can be identified from limited sampled executions, without constructing the complete closed-loop policy graph.

The key observation is that many unsuccessful trajectories can converge to the same persistent failure. For example, an agent may repeatedly attempt an ineffective action at a locked door, even when arriving from very different starting positions. We do not claim that failures originate from a small subset of starts; rather, many starts may share one failure outcome.

For a frozen deterministic policy, each start reaches the goal or enters a non-goal fixed point or cycle. This decomposition is established mathematics. Our question is whether its consequential failure structures can be discovered efficiently as the number of states and goals increases.

2. Why do the sampling metrics matter?

Recovered recurrent failure mass is the share of total recurrent failure probability accounted for by structures found through sampling. It measures whether we find the important failures, not merely how many distinct failures we find.

For MG16, the exact evaluation population contains 32,592 start–goal conditions across 84 right-room goals. Sampling 32 starts per goal identifies structures accounting for 85% of pooled recurrent failure mass. The 98.8% statistic is the fraction of goal–trial pairs in which the dominant recurrent failure is discovered. These are measures of discovery quality against exact ground truth, not general claims about large-scale performance.

In FR12, exact evaluation across five independently trained policies establishes the frequencies of success, fixed-point failures, and cycles. This census supplies ground truth for testing what sampled executions recover; its counts are descriptive, not statistical significance claims.

3. Why is this useful?

A recovered recurrent failure can also identify an intervention target. In our FR12 example, 45 starts flow to the same fixed point. Changing one action there redirects this incoming basin to success, increasing successful starts from 55/124 to 100/124.

The investigation therefore proceeds from structural diagnosis (which starts share a failure), to sampled recovery (how much failure mass is found with limited executions), to actionability (whether the discovered structure helps locate an intervention). The central contribution is sample-efficient diagnosis; intervention tests its utility.

4. Presentation and related work

We agree that the abstract and introduction should lead with the practical question and explain the metrics more plainly. On related work, Section 2, Related Work, follows the Introduction, and Appendix A extends that discussion. These sections cover policy explanation graphs, recurrent dynamics, sampled verification, AEGIS-RL, and failure-guided interventions. Nevertheless, the Introduction itself should situate the question in that literature more clearly, rather than relying on the subsequent sections.

5. Experimental scope

We agree that three continuation seeds cannot establish statistically significant superiority. Seed 0 was developmental; seeds 1–2 followed the fixed protocol. Outcomes are heterogeneous, and guided and random triggering had unequal realized exploration exposure. We therefore treat continuation as exploratory evidence of intervention potential, not a conclusive comparison.

The five-seed FR12 experiments support structural and sampled-recovery findings in their evaluated deterministic setting. They do not establish general scalability to stochastic, continuous, or much larger systems.

In summary, we do not claim new graph mathematics or statistically proven training superiority. We investigate whether consequential recurrent failures can be recovered from limited policy executions and used to guide intervention. We appreciate the reviewer highlighting the need to make that motivation and the scope of the findings clearer.

Thank you for the response. I understand that the authors want to demonstrate how small interventions can lead to impressive improvements in performance in goal-conditioned RL. However, I suggest the authors take some time and think about how they want to present their research. This manuscript, in its current state, fails to communicate the problem or the outcomes of the research to the reader. Your response still doesn't tell me why this is an important problem to solve. Instead of using a benchmark to explain this, the authors should connect their work to prior work so that a reader can understand what gap the paper wants to fill. A paper that is not trying to solve a well motivated research problem is not very interesting. Further, the empirical results are statistically very weak and speak nothing about how the results would hold up on more seeds or different settings. I recommend that the authors work on this as well, for their own understanding of their solution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.