acceptodds
Under review as a conference paper at ICLR 2027

GameConsole: Fine-Grained Gameplay Description with Executable Knowledge Cartridges

Abstract

Fine-grained gameplay description requires identifying game-defined entities and states in observed events. The resources and recognition strategies needed to resolve these details vary across games, making adaptation difficult when game-specific knowledge is tightly coupled with recognition pipelines. We introduce , a console–cartridge framework that separates game-specific resources and recognition bindings from reusable recognition operations. A replaceable knowledge cartridge contains entity definitions, visual references, relations, and target-specific recognition bindings, while a shared console uses these bindings to instantiate recognition programs from a common operator library. These programs combine complementary visual evidence and use game-defined relations to constrain candidate identities. Event-aware sampling allocates observations according to visual change, and the resulting observations are combined with source-linked recognition facts to generate temporally localized descriptions. Across three games and three generation backends, improves mention F1 and reference-conditioned evidence scores over direct and knowledge-enhanced baselines under shared video observations. Target-level comparisons further show that different recognition targets benefit from different operators. Downstream QA on Valorant shows that the generated descriptions improve recovery of game-specific details and temporal retrieval. The code is available here: https://anonymous.4open.science/r/GameConsole-7FD6

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
fine-grained video understanding, game video analysis, VLM, multimodal grounding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KxneOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • Motivation: Fine-grained gameplay understanding requires recognizing game-specific entities, states, and events, but different games—and even different entity types within the same game—often require different knowledge and recognition strategies.
  • Method: The paper proposes GAMECONSOLE, a modular Console–Cartridge framework. The shared Console provides reusable recognition operations such as segmentation, filtering, matching, and post-processing, while each game-specific Cartridge contains entity definitions, visual references, relations, and recognition bindings that specify what to recognize and how to recognize it. This allows adaptation to a new game mainly by replacing the Cartridge while keeping the Console fixed.
  • Temporal Processing: GAMECONSOLE also uses event-aware adaptive sampling, allocating fewer observations to visually stable periods and more around transitions or events. The recognized game-specific facts are linked to their source timestamps and combined with time-aligned visual evidence to generate fine-grained, temporally localized gameplay descriptions.
  • Key Findings: Experiments across three games and three generation backends show that GAMECONSOLE consistently improves fine-grained description quality over direct VLM generation and knowledge-enhanced baselines. The experiments also show that different recognition targets benefit from different visual operators, supporting the paper’s modular design. On Valorant, the generated descriptions further improve downstream tasks such as post-match review, event explanation, and temporal retrieval.
Critical strengths
  1. Novel modular design: The Console–Cartridge framework cleanly separates reusable recognition operations from game-specific knowledge and recognition strategies.
  2. Flexible recognition: Different entity types can use different specialized operators, rather than relying on a single VLM for all recognition tasks.
  3. Strong empirical results: GAMECONSOLE consistently improves fine-grained descriptions across three games and also benefits downstream gameplay QA tasks.
Critical weaknesses
  1. High system complexity: The framework requires game-specific cartridges, visual references, recognition bindings, and potentially specialist models, making adaptation less plug-and-play than the “replaceable cartridge” framing may suggest.
  2. Limited game coverage: Experiments cover only three games, so it remains unclear how well the framework scales to a much broader range of games and visual styles.
  3. Dependence on external recognition components: Performance depends partly on the quality and selection of specialized visual operators, so improvements cannot be attributed solely to the overall framework itself.
Questions for authors
  1. Framework Complexity: Although the Console–Cartridge design aims to simplify adaptation to new games, constructing a game-specific Cartridge still requires entity definitions, visual references, and recognition bindings. How much manual effort and game-specific engineering are required to build a new Cartridge? Could the authors clarify which components can be automatically configured?

  2. Generalization to New Games: The experiments cover only three games. How well would GAMECONSOLE generalize to games with substantially different visual styles, entity types, or gameplay mechanics? Are there any particular types of games where the proposed framework may be less effective?

  3. Contribution of Specialized Operators: Since GAMECONSOLE relies on different specialized visual operators, how much of the performance improvement comes from the Console–Cartridge framework itself versus the capabilities of these external recognition components? Could the authors clarify whether the same operators, when used without the proposed framework, achieve comparable performance?

5 comments

Part 1

We thank the reviewers for their careful reading and detailed feedback. Below we respond to each point in turn. Revised text is marked in blue in the updated manuscript.

W1 / Q1. Manual effort and automatic execution

Building a cartridge involves preparing game resources and configuring existing recognition operators. In our three-game frozen-console study, hands-on configuration, debugging, and quality checks take 0.5–1.1 person-hours. The concrete adaptation procedure is:

  1. Prepare resources. Select the target entities, collect canonical names, relations, and reference images from Wikis/APIs or game resources, and bind their reference paths. Reusable scripts download and organize the selected resources.
  2. Specify when and where to recognize. Define Boolean routing conditions, such as whether the inventory is open, and draw a fixed ROI once for each static interface region. Existing localization operators find dynamic targets such as characters or ability effects.
  3. Configure and check recognition. Enable the required registered programs, select operators and reference sets, and calibrate and inspect their outputs. For example, Don't Starve's inventory and backpack programs share DINOv2–AKAZE matching (P7–P8), while Silksong's HUD and inventory programs reuse DINOv2 matching (P13–P14).

Specialist-data construction is automated after the author defines the targets and prompts. GPT-4o refines the target prompts, locally deployed SAM3 proposes regions and bounding boxes, and batched API verification selects valid proposals. Scripts export accepted annotations into YOLO training data, run training, and index references; authors inspect data quality and recognition outputs (Appendix B.5).

Three game-familiar undergraduates each adapted one additional game—Mahjong, Terraria, or Slay the Spire—with the console, executor, and operator implementations fixed, changing cartridge resources, configuration, and optional specialist weights. Their measured costs are:

Adaptation itemMahjongTerrariaSlay the Spire
Router and ROI configuration (person-hours)0.00.10.1
Binding, debugging, and quality checks (person-hours)0.50.51.0
Total hands-on time (person-hours)0.50.61.1
Knowledge and asset downloading (automated, h)0.51.52.0
Specialist-data (YOLO) preparation (automated, h)1.01.00.0
Automatic proposal generation/filtering (h)0.61.80.0
Specialist training (h)0.20.70.0
Reference indexing (h)0.10.20.2
Sum of automated stage times (h)2.45.22.2
YOLO training entries2,17410,820No new YOLO

Automatic times are sums of stage elapsed hours on eight H20 GPUs; human effort is reported in person-hours. Evaluation annotation is accounted for separately.

For the Valorant preparation example, the pipeline processes 59,007 sampled frames from the gaming-500-hours dataset. SAM3 proposals/API verification take 8.0 h, and character/ability/minimap YOLO training takes 44/90/14 min, totaling approximately 10.5 h for these stages. Preparation uses eight H20 GPUs, with each YOLO run using one GPU; API verification consumes approximately 2.25M input / 30K output tokens. Frame extraction and screening are outside this stage-time total.

Part 2

W2 / Q2. New-game adaptation and recognition benefits

The frozen-console study extends recognition to three additional games with different layouts and entity types. The original evaluation covers tactical shooting (Valorant), survival and crafting (Don't Starve), and 2D exploration and combat (Silksong). The added games cover tile recognition, world entities and inventory items, and cards and status indicators using existing operators.

Both VLM baselines use gpt5-chat. All methods receive the same frames, annotated target ROIs, and canonical candidate sets. This controlled evaluation isolates identity recognition given target regions; the reference-augmented baseline additionally receives five retrieved references from the same bank. Detectors select their highest-confidence prediction within each ROI. Test footage is separated from training/reference sources; configuration and thresholds are frozen before test labels are exposed. Accuracy is micro-averaged top-1 identity accuracy over annotated queries, with abstentions counted as incorrect. The frame counts in the table indicate evaluation coverage; they are not the accuracy denominator, which is the pooled set of annotated target queries within those frames.

Recognition evaluationMahjongTerrariaSlay the Spire
Recognition targetsMahjong tilesCharacters, NPCs, monsters, inventory itemsHand cards, characters, monsters, status indicators
Reused operators/compositionsDetection and local-feature matchingDetection, world-entity and inventory-icon matchingSegmentation, local-feature matching, and DINOv2 icon matching
Test frames (evaluation coverage)89120100
VLM + canonical candidate names (%)87.4041.8053.60
VLM + retrieved references (%)90.6058.2066.40
Adapted GameConsole (%)92.1066.7074.00
Gain over reference-augmented VLM (points)+1.50+8.50+7.60

With the console and operator implementations fixed, the three adapted cartridges improve identity accuracy over the reference-augmented VLM by 1.50, 8.50, and 7.60 points, respectively. This demonstrates reuse across tile, world-entity, and card/interface recognition by changing game resources and recognition bindings.

More challenging cases arise in games where different entities or effects are visually indistinguishable in the available observations. Nearly identical appearances, heavy occlusion, and densely overlapping effects, such as abilities in MOBA combat, can leave insufficient visual cues for reliable identification. Some identities also require motion or longer temporal context to distinguish. Additional localization or temporal-recognition operators may help when those cues are available (§3.1; Appendix B.6). We will discuss these visually ambiguous cases and their implications for recognition in the Limitations section.

Part 3

W3 / Q3. Contribution of specialized operators

We first isolate GameConsole's execution mechanisms while keeping recognition operators fixed. On the same Valorant/Mobile benchmark with gpt5-chat, we retain the operators, pretrained weights, visual references, sampled observations, thresholds, and generation settings, and disable only the specified mechanism:

Fixed-operator configurationEvent-averaged Mention F1 (%) ↑
Without router52.39
Without relation conditioning68.42
Full GameConsole73.30

Removing the router reduces F1 by 20.91 points, while removing relation conditioning reduces it by 4.88 points. Thus, the gains are not attributable only to the availability of specialist operators: scene-appropriate routing and game-defined map–location constraints also contribute under fixed-operator settings. With the same cached recognition predictions, retaining source timestamps, frame/ROI associations, and matched visual references further improves F1 by 5.43 points over a canonical-name-only list (67.87 to 73.30; see the evidence-organization control).

Target-specific recognition and detector roles. For dynamic visual elements, specialist detectors locate the targets and return bounding boxes for cropping and retrieval. For ability effects and minimaps, YOLO supplies the regions, while downstream reference matching determines the specific ability or map identity. The target-specific programs use detection outputs as follows (Table 9):

  • Character identity (P2): The trained identity YOLO provides character bounding boxes and also directly predicts their canonical identities.
  • Ability identity (P3): YOLO locates the visual effect; DINOv3 and visual-feature matching compare that region with game-specific references to identify the ability.
  • Map identity (P4): YOLO locates and crops the minimap; AKAZE matches the crop to map references. Here, YOLO supplies the region, while reference matching supplies the map identity.

The MM-RAG baselines share observation windows, configured ROIs, and visual-reference resources. All methods in this comparison use gpt5-chat, the same event windows, and the same sampled frames. MM-RAG and MM-RAG+ use the same configured ROIs and retrieve from the same cartridge reference bank. MM-RAG embeds ROI crops, retrieves Top-K reference images, and lets the VLM identify the canonical entity or none and generate the description. MM-RAG+ further places the ground-truth-matched reference first in the candidate list. Table 1 reports:

Method (gpt5-chat event-averaged mention F1, %)Valorant / MobileDon't StarveSilksong
MM-RAG43.053.331.3
MM-RAG+46.460.432.7
GameConsole73.362.067.7

GameConsole exceeds MM-RAG+ by 26.9, 1.6, and 35.0 F1 points, respectively. The retrieval baselines have access to the configured target regions and reference images, including an oracle reference in MM-RAG+. GameConsole additionally applies target-specific recognition procedures to resolve those visual observations into canonical identities before description generation; the fixed-operator ablations above isolate execution effects more directly.

Table 2 explains why recognition is configured by target. Here, “Specialist” denotes the complete target-specific recognition program. The table reports recognition scores (%) on GCBench-Prog:

TargetEmbeddingLightGlueSpecialist
Character (P2)30.7962.8184.87
Ability (P3)43.2510.1077.06
Location (P5)24.463.2—

For ability identification, the compared methods receive the same fixed detection boxes, so the comparison measures identification within those boxes. The specialist achieves 77.06, compared with 10.10 for LightGlue; for location recognition, LightGlue achieves 63.2, compared with 24.4 for embedding matching. These within-target comparisons motivate selecting the appropriate recognition procedure for each entity type.

Revision: Specialist terminology and target-specific recognition procedures in §3.2 and Appendix B.3/B.5, with program definitions in Table 9 (p. 17); shared configured ROI/reference inputs in §4.1 and Appendix D.2; operator comparisons and execution controls in §4.3, Tables 2–3 (p. 8).

Reviewer_K5bdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies the game video description/captioning, which requires identifying game-specific entities, states, and event participants. However, traditional works have coupled game-specific resources/ recognition bindings and recognition operations, requiring these pipelines to be reimplemented for each new game.

To this end, inspired by the video game consoles, this work has proposed GameConsole, which separates replaceable game-specific resources and recognition bindings from reusable recognition operations, enabling each cartridge to configure executable recognition programs within a shared operator library. Here, knowledge cartidge carries the game-specific knowledge, and the reusable console carries the control logics.

Experiments are conducted on a newly proposed GCBench dataset.

Critical strengths
  1. This work has found that a painful issue in the current work, namely, the game‑specific knowledge is tightly coupled with recognition pipelines. And thus, a new cross‑game fine‑grained gameplay description task was proposed.

  2. A new console‑cartridge architecture to decouple game‑specific knowledge from shared visual operating logics.

  3. It builds a new GCBench benchmark dataset covering description, segmentation, sampling, and downstream QA evaluation.

  4. Experimental results consistently show clear improvements in mention F1 and evidence scores across three games.

  5. Strong engineering pipeline.

Critical weaknesses
  1. High-cost manual work may be required when creating new knowledge cartridges for unseen games.

  2. It seems that the framework still needs manual configuration; it can not fully auto‑generate cartridges for entirely new games.

  3. Visual‑change‑driven sampling may miss some important events or introduce massive workloads in different types of games or in different settings.

  4. This a engineering work and lacks enough theoretical contribution/novelty.

  5. Besides the manual costs, this pipeline also faces the computational cost issue. This work lacks enough discussion on this point.

  6. The evaluation metrics should be improved. At least some embedding-based metrics should be improved in Table 1.

Questions for authors
  1. The detailed costs of this work.

  2. See my weaknesses.

7 comments

Part 1

We thank the reviewers for their careful reading and detailed feedback. Below we respond to each point in turn. Revised text is marked in blue in the updated manuscript.

W1–W2 and Q1. Manual effort and automatic cartridge construction

Replacing a cartridge means deploying a prepared game-specific package. Adaptation changes canonical entities, visual references, relations, routing/ROI settings, bindings, and optional YOLO weights while reusing the shared operator catalog and executor.

Humans select targets/resources, configure relations/router/ROIs, bind operators, and check programs. Proposal generation, filtering, training, indexing, and recognition then execute automatically. Pseudo-labeling reduces manual bounding-box annotation.

Three game-familiar undergraduates each adapted one additional game—Mahjong, Terraria, or Slay the Spire—with the console, executor, and operator implementations fixed, changing cartridge resources, configuration, and optional specialist (YOLO) weights:

Adaptation itemMahjongTerrariaSlay the Spire
Router and ROI configuration (person-hours)0.00.10.1
Binding, debugging, and quality checks (person-hours)0.50.51.0
Total hands-on time (person-hours)0.50.61.1
Knowledge and asset downloading (automated, h)0.51.52.0
Specialist-data (YOLO) preparation (automated, h)1.01.00.0
Automatic proposal generation/filtering (h)0.61.80.0
Specialist training (h)0.20.70.0
Reference indexing (h)0.10.20.2
Sum of automated stage times (h)2.45.22.2
YOLO training entries2,17410,820No new YOLO

Automatic times are sums of stage elapsed hours on eight H20 GPUs; human effort is reported in person-hours. Evaluation annotation is accounted for separately. Mahjong reuses detection/local-feature matching; Terraria reuses detection and world-entity/inventory-icon matching; Slay the Spire reuses segmentation, local-feature matching, and DINOv2 icon matching.

Recognition evaluationMahjongTerrariaSlay the Spire
Test frames (evaluation coverage)89120100
VLM + canonical candidate names (%)87.4041.8053.60
VLM + retrieved references (%)90.6058.2066.40
Adapted GameConsole (%)92.1066.7074.00
Gain over reference-augmented VLM (points)+1.50+8.50+7.60

Both VLM baselines use gpt5-chat. All methods receive the same frames, annotated target ROIs, and canonical candidate sets. This controlled evaluation isolates identity recognition given target regions; the reference-augmented baseline additionally receives five retrieved references from the same bank. Detectors select their highest-confidence prediction within each ROI. Test footage is separated from training/reference sources; configuration and thresholds are frozen before test labels are exposed. Accuracy is micro-averaged top-1 identity accuracy over annotated queries, with abstentions counted as incorrect. The frame counts in the table indicate evaluation coverage; they are not the accuracy denominator, which is the pooled set of annotated target queries within those frames.

With the console and operator implementations fixed, the three adapted cartridges improve identity accuracy over the reference-augmented VLM by 1.50, 8.50, and 7.60 points, respectively.

Revision: adaptation setting in §3.1 and Appendix B.6; summary in §4.6, Table 7 (p. 10); prospective costs and target-level evaluation in Appendix E.4, Tables 26–27 (p. 41).

Part 2

W3. Sampling robustness and workload

Sampling introduces a measurable quality–cost trade-off. Table 4 compares six policies on the same 119 Valorant events, with event boundaries and generation settings fixed:

SamplingCharacter F1Weapon F1Ability F1Map F1Location F1Mean F1Frames/eventPrompt tokens/eventLatency (s/event)
Uniform 1 s60.294.066.290.068.675.8018.134,30064.8
Uniform 2 s49.384.251.287.863.767.249.320,42943.4
Uniform 3 s50.786.053.185.652.965.666.716,16135.4
Uniform 4 s56.083.345.281.139.661.045.113,47532.9
Uniform 8 s47.076.837.662.227.250.163.110,09923.9
GameConsole60.685.962.187.859.971.265.114,21931.6

At the same mean frame count as Uniform 4 s, GameConsole improves mean F1 by 10.22 points, using 5.5% more prompt tokens. Relative to Uniform 1 s, it reduces prompt tokens by 58.5% and latency by 51.2%, with a 4.54-point F1 decrease. Latency includes recognition and description generation, excluding decoding and sampling.

Human review (Table 23) assigns high tolerance to 106/119 events (89.1%), medium tolerance to 11, and low tolerance to 2. High tolerance means the main event remains usable despite possible detail loss. Configurable temporal profiles let users adjust sampling density to their workload.

Revision: §4.4, Table 4 (p. 8), and Appendix E.2, Table 23 (p. 39), clarify the measured sampling trade-off and its scope.

W4. Methodological contribution

GameConsole studies how game-specific knowledge configures reusable recognition, rather than proposing tool composition alone. Visual references define candidate identities, game-defined relations constrain ambiguous matches (e.g., map–location candidates), and target bindings determine the applicable recognition procedures. Shared execution retains source-linked facts for event description. We build on established pretrained visual operators, score fusion, and MAD normalization.

Frozen-console adaptation evaluates reuse; controlled experiments on the same Valorant/Mobile benchmark with gpt5-chat measure execution and evidence-organization benefits.

Fixed-operator ablations. We keep operators, pretrained weights, visual references, sampled observations, thresholds, and generation settings fixed, disabling only the corresponding execution mechanism:

  • Without router: Disables scene/interface applicability gates while retaining target-type checks, relation-conditioned candidate filtering, and dependency order.
  • Without relation conditioning: Removes game-defined map–location constraints by matching against the full reference bank instead of the predicted map's candidate bank.
Fixed-operator configurationEvent-averaged Mention F1 (%) ↑
Without router52.39
Without relation conditioning68.42
Full GameConsole73.30

Removing the router reduces F1 by 20.91 points, while disabling relation conditioning decreases it by 4.88 points. With recognition operators fixed, these results quantify the benefits of selecting scene-appropriate procedures and using game-defined relations to disambiguate visually similar candidates.

Evidence organization control. Both conditions use the same original observations and cached recognition predictions, with identical generation instructions and decoding settings:

  • Canonical names only: Supplies recognized canonical identities as a flat textual list, without their timestamps, spatial locations, or links to visual references.
  • Source-grouped visual evidence: Supplies the same recognized identities together with source timestamps, frame/ROI associations, and matched visual references.
Presentation of cached outputsEvent-averaged Mention F1 (%) ↑
Canonical names only67.87
Source-grouped visual evidence73.30

With cached recognition predictions held fixed, source-grouped visual evidence improves F1 by 5.43 points over a canonical-name-only list. This comparison evaluates the value of retaining temporal, spatial, and visual-source associations; it is not a format-only comparison of otherwise identical, fully serialized tool outputs.

Revision: the contribution statement in the Introduction and §2–§3.2; controlled comparisons in §4.3, Table 3 (p. 8); relation conditioning and execution details in Appendix B.2–B.3, and the evidence-organization protocol in Appendix D.2 (p. 31).

Part 3

W5 and Q1. Computational cost

Automatic proposal/verification and specialist training take approximately 10.5 h. The Valorant pipeline processes 59,007 sampled frames from the gaming-500-hours dataset (Appendix B.5; Table 12; preparation times in Appendix E.5, Table 29):

One-time specialist preparationElapsed time
SAM3 proposals and parallel API verification8.0 h
Character YOLO training44 min
Ability YOLO training90 min
Minimap YOLO training14 min
Approximate sum of the listed automated stage times≈10.5 h

Preparation uses eight H20 GPUs, with each YOLO run using one GPU. API verification consumes approximately 2.25M input / 30K output tokens. The table sums proposal/verification and training elapsed times; frame extraction and screening are outside this total.

For inference, matched runs on 30 videos (1,100 frames) use eight H20 GPUs and identical inputs. Persistent processes and conditional execution reduce total wall-clock time from 5,638 to 2,892 s (1.95×; 48.7% reduction):

ExecutionTotal time (s)Time per sampled frame (s)
Before optimization5,6385.13
With execution optimizations2,8922.63

Timing covers initialization, routing, and local recognition; decoding and API-based caption generation are outside this measurement.

We also compare Direct and GameConsole on 60 Valorant/Mobile events, fixing event windows, sampled base frames (mean 5.0/event), gpt5-chat, output cap, and API concurrency of one:

MethodCaption prompt tokens/eventRecognition + routing (s/event)Caption API (s/event)Total (s/event)
Direct9,1060.016.416.4
GameConsole14,08711.220.731.9

Values are arithmetic means from uncached live API calls, including request wall time. Recognition includes router API calls and precedes caption generation; decoding, sampling, and one-time preparation are outside the timing. GameConsole adds 15.5 s/event in this setting.

Revision: preparation accounting in Appendix B.5 and E.5, Table 29 (p. 42); runtime summary in §4.6; runtime definitions and paired costs in Appendix E.5, Tables 28–30 (p. 42).

W6. Embedding-based evaluation

We added BERTScore F1 using bert-base-chinese, alongside canonical-mention F1 and Evid. Scores below are multiplied by 100:

GameMethodgpt5-chatgpt-5.6-terraglm-5.3-flash
Valorant / MobileDirect71.0270.9770.55
Valorant / MobileText KB71.6671.0971.06
Valorant / MobileMM-RAG71.3371.1770.79
Valorant / MobileMM-RAG+71.5671.4471.01
Valorant / MobileGameConsole79.5777.8176.47
Don’t StarveDirect75.3374.9374.05
Don’t StarveText KB75.6074.7073.61
Don’t StarveMM-RAG75.8475.2274.18
Don’t StarveMM-RAG+76.3175.6574.66
Don’t StarveGameConsole81.0579.4977.24
SilksongDirect76.4275.9773.82
SilksongText KB76.7276.1474.06
SilksongMM-RAG76.4876.1073.77
SilksongMM-RAG+76.6276.2573.87
SilksongGameConsole80.0877.2375.47

GameConsole has the highest reported BERTScore in all nine game–generator combinations, with gains of 0.98–7.91 points over the strongest baseline. BERTScore complements canonical-mention F1 and Evid. by measuring contextual semantic similarity alongside entity identification and factual/event consistency.

Revision: complementary semantic-similarity evaluation in §4.1–§4.2 and Appendix D.2, Table 21 (p. 31).

Reviewer_e1z2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates fine-grained gameplay description, which involves generating temporally localized event descriptions that identify game-defined entities, states, and participant roles. Its central idea is a console–cartridge separation. The game-specific cartridge encapsulates knowledge about a particular game, including entity definitions, visual references, entity relations, and bindings that specify the recognition operators to be used for each target. In contrast, the console is shared across games and maintains a library of reusable recognition operators, which are composed into recognition programs based on the cartridge's configuration. The paper claims that, under this design, adapting to a new game requires only replacing the cartridge, while leaving the console unchanged. The recognized facts are subsequently combined with sampled frames and provided to a VLM to generate event descriptions. Experimental results demonstrate that GameConsole outperforms baseline methods across multiple tasks, including fine-grained gameplay description, temporal segmentation, and downstream QA.

Critical strengths

S1. The problem is practically relevant. General-purpose VLMs tend to reduce specific abilities and locations in gameplay to generic terms such as "an ability" or "somewhere". Recovering these details is useful for post-match review, event explanation, and temporal retrieval.

S2. The console–cartridge design is clear and practical. Separating game-specific knowledge from reusable recognition operators is a sensible abstraction, and relation conditioning (e.g., using the recognized map to restrict location candidates) is an intuitive way to inject game knowledge into recognition.

S3. The main experiment spans multiple games and generators. It covers three games with quite different gameplay and three generators, and compares against four baselines, including the oracle-candidate diagnostic MM-RAG+, showing that the framework can be instantiated on different games. The reported gains are also large; with gpt5-chat, F1 rises from 41.2 to 73.3 on Valorant and from 29.7 to 67.7 on Silksong.

Critical weaknesses

W1. Potentially biased reference descriptions. The GCBench-Desc references were obtained by human review of model-generated drafts (App. D.1), but the paper does not state which pipeline produced the drafts or whether they used recognition outputs. The example draft in Fig. 5 already names precise locations and abilities, which Direct rarely recovers (Fig. 7), so the drafts may come from the proposed method itself, which would inflate the main results in Table 1.

W2. Unsupported adaptation claim. The paper's core motivation is that adapting to a new game only requires replacing the cartridge, but it reports no adaptation cost and no held-out game. In practice, each game requires substantial effort, including per-game recognition programs, router prompts, ROIs, and thresholds, as well as three YOLO detectors trained on pseudo-labels for Valorant (Tables 7, 8, 10; App. B.4–B.5).

W3. Unclear source of the gains. Compared with the baselines, GameConsole adds many stronger recognition tools (e.g., trained detectors for Valorant and full-frame matching), so its gains may come from these tools rather than from the console–cartridge design itself. Ablations that remove the proposed mechanisms (e.g., relation conditioning or the router) and a baseline that feeds the same tool outputs to the VLM as plain text are needed to separate the two.

W4. Limited methodological novelty. As stated below Eq. 2, instantiation "neither trains new modules nor generates arbitrary code"; the framework is essentially a configuration-driven registry of mostly pretrained operators, and components such as score fusion and MAD normalization are standard.

W5. Evaluation limited to a self-constructed benchmark. All experiments (Tables 1–5) are conducted on GCBench, which the authors built themselves, while existing gameplay benchmarks cited in the paper, such as GameplayQA and EgoEsportsQA, are not used. Results on such external benchmarks would make the claims more convincing, especially given the concerns about reference construction in Weakness 1.

W6. Incomplete discussion of related work. The paper omits closely related tool-registry systems such as [1, 2] and the gameplay video captioning dataset [3].

[1] Shen Y, Song K, Tan X, et al. HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face. Advances in Neural Information Processing Systems (NeurIPS), 2023.

[2] Lu P, Peng B, Cheng H, et al. Chameleon: Plug-and-play compositional reasoning with large language models. Advances in Neural Information Processing Systems (NeurIPS), 2023.

[3] Tanaka T, Simo-Serra E. LoL-V2T: Large-scale esports video description dataset. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2021: 4557-4566.

Questions for authors

Q1. [Clarification] Which system produced the drafts for GCBench-Desc, and did it use recognition results? What were the edit rate and inter-annotator agreement during review?

Q2. [Clarification] How much effort (annotation, training, and configuration) did it take to build each cartridge?

Q3. [Suggestion] How do the results in Table 1 change on a reference subset written independently of any model drafts?

Q4. [Suggestion] How much do the results drop when relation conditioning or the router is removed, and how does a baseline that feeds the same tool outputs to the VLM as plain text perform?

Q5. [Future work] Adapt the framework to a new game with the console frozen and report the effort required.

10 comments

Part 1

We thank the reviewers for their careful reading and detailed feedback. Below we respond to each point in turn. Revised text is marked in blue in the updated manuscript.

1. W1 / Q1 / Q3 — Reference provenance and independent validation

GameConsole generated the Valorant/Mobile and Silksong drafts using its recognition outputs. Don't Starve combined pre-existing human-authored references with an additional reviewed set, aligned against sampled observations. Human reviewers check factual correctness, completeness, and temporal organization, correcting unsupported claims and adding identifiable missing facts.

We re-evaluate fixed predictions against independent references. Before inspecting predictions, we sampled 50 distinct events per game by sampling source videos uniformly and then sampling events within those videos. Two game-familiar annotators independently describe each event from the same observations and canonical terminology guide, without original drafts or evaluated predictions. Disagreements are adjudicated; agreement uses Appendix D.2's canonical-mention matching.

Independent-reference studyValorant / MobileDon't StarveSilksongOverall
Double-annotated events505050150
Inter-annotator canonical-mention F1 (%)95.290.189.991.7

The gpt5-chat predictions and baseline identities remain fixed across reference sets: Text KB for Valorant/Mobile and MM-RAG+ for Don't Starve/Silksong, selected from original full-pool results. F1 is event-averaged; Evid. uses the existing blinded, reference-conditioned 1–5 rubric.

GameReferencesBaseline F1GameConsole F1F1 gainBaseline Evid.GameConsole Evid.
Valorant / MobileOriginal subset50.772.1+21.42.743.52
Valorant / MobileIndependent51.171.9+20.82.773.49
Don't StarveOriginal subset61.063.4+2.43.273.42
Don't StarveIndependent60.562.7+2.23.243.39
SilksongOriginal subset32.967.3+34.42.763.50
SilksongIndependent34.166.5+32.42.793.46

With independent references, GameConsole retains F1 advantages of 20.8, 2.2, and 32.4 points for Valorant/Mobile, Don't Starve, and Silksong, respectively.

The original Valorant/Mobile review audit reports:

Original review outcomeEventsShare of 3,712 events
Human-reviewed3,712100.0%
Revised1,87850.6%
Approved without changes1,83449.4%
Entity edit among revised eventsEventsShare of 1,878 revised events
At least one addition92949.5%
At least one removal1,60385.4%
Both additions and removals85445.5%

Edit categories overlap. The 91.7% inter-annotator canonical-mention F1 comes from the new independent annotations.

Revision: reference provenance and independent validation in §4.1 and Appendix D.1–D.2; the independent-reference results are in Tables 16–17 (p. 27).

Part 2

2. W3 / Q4 — Fixed-Operator Ablations and Evidence Organization

We measure the contributions of execution mechanisms and evidence organization through controlled experiments on the same Valorant/Mobile benchmark with gpt5-chat.

Fixed-operator ablations. We keep operators, pretrained weights, visual references, sampled observations, thresholds, and generation settings fixed, disabling only the corresponding execution mechanism:

  • Without router: Disables scene/interface applicability gates while retaining target-type checks, relation-conditioned candidate filtering, and dependency order.
  • Without relation conditioning: Removes game-defined map–location constraints by matching against the full reference bank instead of the predicted map's candidate bank.
Fixed-operator configurationEvent-averaged Mention F1 (%) ↑
Without router52.39
Without relation conditioning68.42
Full GameConsole73.30

Removing the router reduces F1 by 20.91 points, while disabling relation conditioning decreases it by 4.88 points. With recognition operators fixed, these results quantify the benefits of selecting scene-appropriate procedures and using game-defined relations to disambiguate visually similar candidates.

Evidence organization control. Both conditions use the same original observations and cached recognition predictions, with identical generation instructions and decoding settings:

  • Canonical names only: Supplies recognized canonical identities as a flat textual list, without their timestamps, spatial locations, or links to visual references.
  • Source-grouped visual evidence: Supplies the same recognized identities together with source timestamps, frame/ROI associations, and matched visual references.
Presentation of cached outputsEvent-averaged Mention F1 (%) ↑
Canonical names only67.87
Source-grouped visual evidence73.30

With cached recognition predictions held fixed, source-grouped visual evidence improves F1 by 5.43 points over a canonical-name-only list. This comparison evaluates the value of retaining temporal, spatial, and visual-source associations; it is not a format-only comparison of otherwise identical, fully serialized tool outputs.

Revision: controlled comparisons in §4.3, Table 3 (p. 8); relation conditioning and execution details in Appendix B.2–B.3, and the evidence-organization protocol in Appendix D.2 (p. 31).

Part 3

3. W2 / Q2 / Q5 — Adaptation effort with the console frozen

Replacing a cartridge means deploying a prepared game-specific package. Humans select targets/resources, configure relations/router/ROIs, bind existing operators, and check programs. Proposal generation, filtering, training, indexing, and recognition then execute automatically through the shared console.

Three game-familiar undergraduates each adapted one additional game with the console, executor, and operator implementations fixed, changing only cartridge resources, configuration, and optional specialist weights. We separate labor from automatic processing:

Adaptation itemMahjongTerrariaSlay the Spire
Knowledge and asset downloading (automated, h)0.51.52.0
Router and ROI configuration (person-hours)0.00.10.1
Binding, debugging, and quality checks (person-hours)0.50.51.0
Specialist-data preparation (automated, h)1.01.00.0
Automatic proposal generation/filtering (h)0.61.80.0
Specialist training (h)0.20.70.0
Reference indexing (h)0.10.20.2
YOLO training entries2,17410,820No new YOLO

Automatic times are sums of stage elapsed hours on eight H20 GPUs; human effort is reported in person-hours. Evaluation annotation is accounted for separately. Mahjong reuses detection/local-feature matching; Terraria reuses detection and world-entity/inventory-icon matching; Slay the Spire reuses segmentation, local-feature matching, and DINOv2 icon matching.

Held-out target identity recognition improves after adaptation. Both gpt5-chat baselines and GameConsole receive the same frames, annotated target ROIs, and canonical candidate sets; the reference-augmented baseline also receives five retrieved references from the same bank. Detectors select their highest-confidence prediction within each ROI. Test footage is separated from training/reference sources; configuration and thresholds are frozen before test labels are exposed. Accuracy is micro-averaged top-1 identity accuracy over annotated queries, with abstentions counted as incorrect. The frame counts in the table indicate evaluation coverage.

Recognition evaluationMahjongTerrariaSlay the Spire
Test frames (evaluation coverage)89120100
VLM + canonical candidate names (%)87.4041.8053.60
VLM + retrieved references (%)90.6058.2066.40
Adapted GameConsole (%)92.1066.7074.00
Gain over reference-augmented VLM (points)+1.50+8.50+7.60

With the console and operator implementations fixed, the three adapted cartridges improve identity accuracy over the reference-augmented VLM by 1.50, 8.50, and 7.60 points, respectively.

Valorant specialist preparation processes 59,007 sampled frames from the gaming-500-hours dataset. Automatic proposal/verification and training take approximately 10.5 h:

One-time automatic preparation stageElapsed time
SAM3 proposals and parallel API verification8.0 h
Character / ability / minimap YOLO training44 / 90 / 14 min
Approximate sum of the listed automated stage times≈10.5 h

Preparation uses eight H20 GPUs, with each YOLO run using one GPU. API verification consumes approximately 2.25M input / 30K output tokens. The table sums proposal/verification and training elapsed times; frame extraction and screening are outside this total.

Revision: adaptation scope in §3.1 and Appendix B.6; adaptation summary in §4.6, Table 7 (p. 10); prospective costs and recognition evaluation in Appendix E.4, Tables 26–27 (p. 41); specialist preparation in Appendix B.5 and E.5, Table 29 (p. 42).

Comments on the paper

1 comment

The problem. Show a strong VLM a Valorant clip and it says "a player prepares an ability." A player would say "KAY/O lines up ZERO/POINT on Icebox." The missing piece is game-specific knowledge, and it changes with every game. We study fine-grained gameplay description as a cross-game task. GameConsole puts each game's knowledge into a swappable cartridge, and a shared console turns it into recognition programs from one operator library. To support a new game, swap the cartridge, not the console.

Where reviewers stand.

  • e1z2 raised their score to Weak accept, saying the new experiments address most of my main concerns. Their three remaining points are answered below.
  • Kxne lists the novel modular design as a key strength (no post-rebuttal reply).
  • K5bd's final comment: It looks like time consumption is still too high. See our note at the end.

Concerns and how we resolved them. ✓ = raised (weakness no.) · ★ = cited as a strength · Solved = reviewer confirmed it, did not raise it again, or we answered the follow-up.

Concerne1z2K5bdKxneOur evidenceReviewer feedback
Reference bias: is the referee also a player?✓ (W1)150 events re-annotated from scratch, blind to drafts and predictions (IAA 91.7%). Gains hold: +20.8 / +2.2 / +32.4 F1Solved: e1z2 independent-reference check; see (i)
Novelty: is this more than a pipeline?✓ (W4)✓ (W4)★ strengthGame knowledge configures and constrains reusable operators (e.g., the recognized map narrows location candidates), and every fact keeps its source linkSolved: not raised again by e1z2 or K5bd
Source of gains: is it the console, or just strong operators?✓ (W3)✓ (W3)With operators fixed: no router −20.9 F1, no relation conditioning −4.9, flat name list −5.4. MM-RAG+ gets the same YOLO boxes and an oracle reference, yet trails by 26.9 / 1.6 / 35.0 F1Solved: e1z2 fixed-operator ablations
Manual effort: what does a new cartridge cost?✓ (W2)✓ (W1, W2)✓ (W1)Console frozen, three new games: 0.5–1.1 person-hours of hands-on work, plus 2.2–5.2 h automatedSolved: e1z2 frozen-console adaptation
Generalization: does it work beyond three games?✓ (follow-up)✓ (W2)Mahjong / Terraria / Slay the Spire, with end-to-end benchmarks annotated without GameConsole: F1 +2.2 / +11.8 / +6.1, Evid. +0.13 / +0.42 / +0.31 over the strongest baselineSolved: e1z2's follow-up (iii) answered by these results
Runtime: is it too slow?✓ (W3, W5)Recognition 5.13 → 2.63 s/frame; +15.5 s/event vs. Direct; see note belowResponded, but no follow-up
External benchmark: why not GameplayQA or EgoEsportsQA?✓ (W5)Different task (multiple choice, no entity naming); EgoEsportsQA has no public release. See (iii)Responded, but no follow-up
Metrics: is mention F1 enough?✓ (W6)Added BERTScore: best in 9/9 game × generator settings (+0.98–7.91)Solved: not raised again by K5bd
e1z2's follow-up points.
  • (i) Original and independent references score surprisingly close. Mention F1 counts canonical entities, not wording, so differently worded references naming the same entities get the same score. The 49.4% "approved without changes" counts events needing no factual fix, not shared text.
  • (ii) Automatic prep seems surprisingly short. We ran SAM3 proposals and API verification in parallel on 16 H20 GPUs across two servers with high API rate limits, so Terraria's 10,820 entries took 1.8 h of wall-clock time.
  • (iii) the lack of evaluation on any external public benchmark. GameplayQA and EgoEsportsQA test multiple-choice reasoning (e.g., action order) that never requires naming the agent, ability, or location, the very details our task measures. EgoEsportsQA also abstracts these away by design and has no public release (searched Google, GitHub, Hugging Face, authors' pages).

A note on K5bd's time consumption is still too high.

  • Subjective. The comment says the cost looks too high but names no budget or bottleneck and does not engage with our measurements.
  • Already optimized. Persistent processes and conditional execution cut recognition time 48.7% (5.13 → 2.63 s/frame); adaptive sampling halves latency vs. 1 fps sampling (64.8 → 31.6 s/event, −4.5 F1). The remaining +15.5 s/event over Direct buys +32.1 F1 on Valorant, and the one-time cartridge cost (≤1.1 person-hours per game) is reused for every video.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.