- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes Fira: an inference-time intervention for RL agent policies. Fira reasons over a graph representation of the scene to decide when and how to intervene on the base policy with a "rescue" action. The approach is validated on a serval video games benchmarks including Kangaroo, Seaquest, Frostbite and VizDoom.
Critical strengths
None
Critical weaknesses
- The introduction is quite vague when motivating crucial aspects of Fira:
"retraining for each such failure is often impractical, particularly when failures surface only after deployment."
In terms of compute cost? In terms of cost of interacting with the environment? It is difficult for the reviewer to evaluate how much "cost" Fira saves without explicitly knowing what costs are being considered. The type of cost should be clearly stated and have supporting experiments showing what "cost" Fira reduces (and by how much!).
Fortunately, many share a common structure: a condition in the current scene signals an impending failure, and a targeted corrective action is needed only while that condition holds.
This is probably true in simple (mostly markov) video games. Is it true in other domains? Is Fira supposed to work beyond such beyond simple video game settings?
-
The work's core premise is that inference-time interventions are useful because retraining the policy is expensive. The reviewer agrees that this could be true (either due to compute required to train the policy or expense of data collection through environment interaction). However, neither is true in the considered setting: training a small policy network on a video game is cheap in terms of both compute and interaction (and the proposed approach seems constrained to this setting). For example, Fira's dataset collection would be equivalent to collecting a dataset of car crashes in the autonomous driving setting---a very expensive task.
-
The evaluations are limited to simple games, and leverage an array of "privileged accesses" unique to this setting. Overall, the proposed approach seems unlikely to work beyond the simple video game setting.
-
The policy being patched is quite simple and weak, it would be interesting to see if Fira can help more complex decision policies (such as those produced by frontier LLMs) and/or more current RL game agents [1,2].
-
The writing is quite poor and needlessly obtuse when presenting simple ideas.
[1] S. Fujimoto, P. D’Oro, A. Zhang, Y. Tian, and M. Rabbat, “Towards General-Purpose Model-Free Reinforcement Learning,” Jan. 2025, ICLR 2025, doi: 10.48550/arxiv.2501.16142.
[2] D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba, “Mastering Atari with Discrete World Models,” Oct. 2020, ICLR 2021, doi: 10.48550/arxiv.2010.02193.
Questions for authors
Please see the above weaknesses section.