acceptodds
Withdrawn from ICLR 2027

FIRA: DEPLOYMENT-TIME RESCUE OF FROZEN RL AGENTS VIA A LOGICAL FIREWALL

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Frozen reinforcement-learning policies can fail at deployment in narrow, recognizable ways, yet retraining them may be impractical. We study deployment-time rescue: correcting a frozen policy with a separately trained intervention layer, without modifying the policy or adapting during deployment. Training such a layer offline exposes an offline-to-live rescue gap: models with similar offline accuracy can behave very differently in closed loop, and running the rule-based supervision procedure as a live controller does not predict how well the learned models will perform. We introduce FIRA (Firewalled Intervention via Relational Atoms), which represents scenes as ego-centric graphs of affordance-tagged entities and predicts atoms describing threats, resources, and progress. A logical firewall blocks action-loss gradients through the predicted-atom pathway and permits the rescue action to override the frozen policy only when an intervention atom fires. FIRA is trained on fixed offline scenes labeled from environment rules, without reward optimization, rescue-policy rollouts, or deployment-time adaptation. Under identical supervision, matched flat-MLP and behavior-cloning baselines attain high offline accuracy but transfer less consistently to closed-loop control. Across three Atari games, two frozen-policy families, and two ViZDoom scenarios, FIRA rescues novel hazards, resource mismanagement, reactive-control failures, unseen threats, and health depletion. On Frostbite, FIRA raises the frozen policy's score from 126 to 2527 and advances beyond the first level in 19 of 20 episodes, compared with 0 of 20 without rescue.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, deployment-time robustness, policy correction, neuro-symbolic learning, relational reasoning, graph neural networks

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kYxJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Fira: an inference-time intervention for RL agent policies. Fira reasons over a graph representation of the scene to decide when and how to intervene on the base policy with a "rescue" action. The approach is validated on a serval video games benchmarks including Kangaroo, Seaquest, Frostbite and VizDoom.

Critical strengths

None

Critical weaknesses
  • The introduction is quite vague when motivating crucial aspects of Fira:

"retraining for each such failure is often impractical, particularly when failures surface only after deployment."

In terms of compute cost? In terms of cost of interacting with the environment? It is difficult for the reviewer to evaluate how much "cost" Fira saves without explicitly knowing what costs are being considered. The type of cost should be clearly stated and have supporting experiments showing what "cost" Fira reduces (and by how much!).

Fortunately, many share a common structure: a condition in the current scene signals an impending failure, and a targeted corrective action is needed only while that condition holds.

This is probably true in simple (mostly markov) video games. Is it true in other domains? Is Fira supposed to work beyond such beyond simple video game settings?

  • The work's core premise is that inference-time interventions are useful because retraining the policy is expensive. The reviewer agrees that this could be true (either due to compute required to train the policy or expense of data collection through environment interaction). However, neither is true in the considered setting: training a small policy network on a video game is cheap in terms of both compute and interaction (and the proposed approach seems constrained to this setting). For example, Fira's dataset collection would be equivalent to collecting a dataset of car crashes in the autonomous driving setting---a very expensive task.

  • The evaluations are limited to simple games, and leverage an array of "privileged accesses" unique to this setting. Overall, the proposed approach seems unlikely to work beyond the simple video game setting.

  • The policy being patched is quite simple and weak, it would be interesting to see if Fira can help more complex decision policies (such as those produced by frontier LLMs) and/or more current RL game agents [1,2].

  • The writing is quite poor and needlessly obtuse when presenting simple ideas.


[1] S. Fujimoto, P. D’Oro, A. Zhang, Y. Tian, and M. Rabbat, “Towards General-Purpose Model-Free Reinforcement Learning,” Jan. 2025, ICLR 2025, doi: 10.48550/arxiv.2501.16142.

[2] D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba, “Mastering Atari with Discrete World Models,” Oct. 2020, ICLR 2021, doi: 10.48550/arxiv.2010.02193.

Questions for authors

Please see the above weaknesses section.

Reviewer_HmBYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FIRA adds a rescue model to a frozen RL policy. It reads a graph of objects, predicts conditions such as low oxygen or a nearby threat, and takes over when needed. The model learns from scenes labeled by task rules, without retraining the original policy or adapting during deployment.

Critical strengths
  1. Frostbite is a good example: the combined system scores 2,527, compared with 126 for the frozen policy and 30 for the rescue model alone. It also completes the first level in 19 of 20 episodes. This shows that switching between the two controllers can be useful, even when neither works well alone.

  2. The paper includes learned baselines using the same inputs and labels, rules run directly as controllers, and additional held-out evaluation seeds. It also reports cases where simple rules work just as well or better. These comparisons give a clearer picture of what the method actually adds.

Critical weaknesses

The evidence for the firewall is less convincing than the overall rescue results. Table 3 tests gradient detachment only on standalone Kangaroo control, where the intervention gate is not used. The non-detached model also has slightly higher aggregate atom accuracy: 97.2% versus 97.0%. The reward difference suggests that detachment helps training, but it does not directly show that it makes intervention decisions more reliable. That claim needs a comparison with the gate enabled.

Questions for authors
  1. Could you compare models trained with and without detachment in gated mode, using the same data and seed protocol? Reporting missed and unnecessary interventions alongside return would directly test the firewall’s intended benefit.

  2. Do FIRA and the learned baselines vary initialization, data order, and data splitting in the same way?

Reviewer_sGeqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The submission studies deployment-time rescue: correcting a frozen RL policy (PPO+CNN or BlendRL) with a separately trained intervention layer, without updating the base policy or adapting at test time. The proposed layer, FIRA, builds an ego-centric star graph from an object-centric scene description and runs three rounds of message passing into the ego node. The scene description holds entity geometry plus one of four affordance tags, read from simulator object state through a hand-written per-task adapter. Two heads predict per-game binary predicates ("atoms") and a corrective action, and the action head reads the atoms through a stop-gradient. At deployment, the rescue action replaces the frozen policy's action whenever an atom from a hand-picked intervention subset exceeds τ = 0.5. The stop-gradient and the gate together are called a "logical firewall." Training is supervised, on fixed offline scenes generated per task by perturbing one scene factor and labeling the result with hand-written environment rules. The submission also frames an "offline-to-live rescue gap": rescue models with similar offline accuracy can differ widely in closed-loop reward, and the rule labeler's live score does not predict how the learned models perform. Experiments cover Kangaroo, Seaquest, Frostbite (JAXAtari) and two ViZDoom scenarios. FIRA is compared with flat-MLP and behavior-cloning rescue models trained on the same labels (Table 1) and evaluated as a gated rescue layer over the frozen policies (Table 2; Frostbite goes from 126 to 2,527). Parts of the firewall are ablated on Kangaroo (Table 3), and standalone FIRA is compared with policies trained live on Kangaroo (Table 4).

Critical strengths
  1. A useful goal. A small add-on that corrects a frozen policy without touching its weights is a worthwhile target, and the design keeps each override traceable to a named atom (Fig. 5). This framing is the main reason the reviewer sees value in the direction.
  2. Candid, up-front reporting. The experimental details are laid out for the reader. The submission gives per-checkpoint results (Table 1), holds out half of the evaluation seeds until design choices were fixed (l.293–296), and pre-declares the checkpoint for each Table 2 row. It also states its limitations openly: privileged input (l.159), settings where the rules alone suffice (l.320–323), and ablation evidence from a single game (l.421). Several of the concerns below can be checked directly against these numbers, which is to the submission's credit.
Critical weaknesses
  1. The positioning and novelty of the contributions are unclear.

The submission calls the offline-to-live rescue gap "the analogue of covariate shift in imitation learning" (l.86-87) and cites the standard references (l.66-67). From an imitation-learning perspective, the gap as presented reads as the familiar open-loop versus closed-loop mismatch, and the main text does not establish what is new about it. One of the two differences the submission gives is that the rescue layer "acts on only a minority of frames" (l.68). Table 2 reports intervention rates of 44.4% (Kangaroo, PPO+CNN) and 66.9% (ViZDoom defend-the-center), and l.423 gives a range of 3-67%. Once defined (l.224-238), the logical firewall is a stop-gradient on concept predictions, which the submission itself relates to sequential concept-bottleneck training (l.233–-234), plus a fixed threshold on hand-specified atoms (Eq. 9). The reasoner is message passing on a star graph (Eqs. 3-4).

  1. The evidence does not support the central claims.

The stated purpose of the stop-gradient is to keep the atoms trustworthy for the gate (l.228–233). In Table 3, offline atom accuracy is 97.2% without the stop-gradient and 97.0% with it, and the ablation runs standalone, where the gate is never used. FIRA is itself trained with supervised losses on a fixed dataset, with no rescue-policy rollouts (l.247–248). The submission does not explain why the gap it identifies leaves FIRA unaffected. Fig. 4a does not show the models at similar offline accuracy. FIRA checkpoints sit near 96–100% and behavior cloning near 91–95%. On Frostbite, the abstract's headline result, the rule-based labeler run live scores 2,582, against 2,527 for FIRA gated (Fig. 4b). The baselines in Table 1 are author-built variants, and Table 2 compares only the frozen policy with and without FIRA

  1. The problem setting and its scope are not justified.

That retraining is impractical is asserted without support (l.54–55). The method depends on privileged object-centric state (l.159) and on per-task hand engineering: a scene adapter, a scene source, a rule labeler, and intervention-atom margins "checked manually against the labeler live" (l.290–293). The deployed gate also departs from Eq. 9: Seaquest overrides also require oxygen below 0.2 (Table 2 caption). Only two frozen-policy families on discrete-action games are tested, and §3 is written around specific Atari observations (oxygen, ammunition, sprites; l.150–151). The scope beyond these settings is unclear.

Exposition: In the reading of the reviewer, the submission comes across as a dense assembly of results, implementation details and pre-built figures, not an argument developed for the reader. Terms appear before their definitions, the figures are not woven into the text, and names shift between figure, caption and table. A new reader, and the community more broadly, needs intentional exposition, a clear position relative to prior work and a logical flow from problem to method to evidence in order to understand and build on a paper. That effort is largely missing from the current version, and its absence made the contribution hard to assess.

Questions for authors
  • What distinguishes the offline-to-live rescue gap from covariate shift in imitation learning, given intervention rates of up to 67% (Table 2, l.423)?
  • Offline atom accuracy is essentially unchanged without the stop-gradient (97.2% vs. 97.0%, Table 3). What evidence in the main paper supports its role in keeping the atoms reliable for the gate?
  • On Frostbite, what does FIRA add over the labeler run live (2,582 vs. 2,527, Fig. 4b)?

Minor notes. These did not decide the recommendation, but together they make the submission hard to read.

  • Terms appear before they are defined in the abstract and introduction: atoms, intervention atoms, relational atoms, logical firewall, rescue situations, shared rescue mechanism (l.57, 61, 74–84). The atom set 𝒜 in Eq. 6 is never defined.
  • Fig. 2 embeds Eqs. 1–4 with symbols that the figure and caption leave undefined (φ_type, φ_tag, I_game, ‖, d_j, air_j, prox_j). The figure is cited once (l.142) and plays no part in the §3.1–3.3 narrative. The Eq. 2 symbols are not defined in the text either.
  • The text in Fig. 3 is too small to read at print size.
  • Fig. 4 combines four visual encodings and two control modes on an axis normalized per setting. Panel (b) prints raw values on normalized bars, and its settings differ from those of Table 1, which supports the same claim.
  • In Table 2, "Reward" and "Success" mean different things from row to row (levels, fractions, diver counts, steps). The ± is a std over seeds in the Atari rows and over checkpoints in the ViZDoom rows, and IR is not defined in the caption.
  • The same ViZDoom failure has three names: "resource recovery" (Fig. 1 header), "resource neglect" (Fig. 1 caption) and "health depletion" (Table 2).
  • The loss terms appear only in passing (l.244–245), and λ is given only in the appendix.
  • The neuro-symbolic RL paragraph (l.99–107) is mostly a list of citations.
  • Details needed to follow the main text sit in a roughly 10-page appendix (A to Q).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.