GameConsole: Fine-Grained Gameplay Description with Executable Knowledge Cartridges
Abstract
Fine-grained gameplay description requires identifying game-defined entities and states in observed events. The resources and recognition strategies needed to resolve these details vary across games, making adaptation difficult when game-specific knowledge is tightly coupled with recognition pipelines. We introduce , a console–cartridge framework that separates game-specific resources and recognition bindings from reusable recognition operations. A replaceable knowledge cartridge contains entity definitions, visual references, relations, and target-specific recognition bindings, while a shared console uses these bindings to instantiate recognition programs from a common operator library. These programs combine complementary visual evidence and use game-defined relations to constrain candidate identities. Event-aware sampling allocates observations according to visual change, and the resulting observations are combined with source-linked recognition facts to generate temporally localized descriptions. Across three games and three generation backends, improves mention F1 and reference-conditioned evidence scores over direct and knowledge-enhanced baselines under shared video observations. Target-level comparisons further show that different recognition targets benefit from different operators. Downstream QA on Valorant shows that the generated descriptions improve recovery of game-specific details and temporal retrieval. The code is available here: https://anonymous.4open.science/r/GameConsole-7FD6
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.