acceptodds
Under review as a conference paper at ICLR 2027

RIGen: Beyond Action Following with Reciprocal Interaction in Generative Gameplay

Abstract

Generative gameplay requires more than immediate responses to player actions. Actions should have persistent consequences, while the generated world should act according to its own state and rules, shaping the player's situation and subsequent choices. Video generation captures local dynamics, but limited history, partially observed game state, and the need for contextual behavior decisions make such interactions difficult to sustain through audiovisual prediction alone. We introduce \rig (Reciprocally Interactive Generation), a framework for reciprocal interaction in generative gameplay. \rig combines persistent state and configurable rules, contextual interaction reasoning, and action-conditioned audiovisual generation. Together, these components make interaction consequences and actor behavior jointly determine generated content while maintaining consistency between game state and visible outcomes. We construct a Xonotic gameplay dataset, train the corresponding audiovisual generator, and develop \rigarena, a playable generative first-person shooter (FPS) game in a fixed arena environment, through post-training. Experiments demonstrate more reliable combat outcomes and reciprocal interaction between the player and the generated world. We will release all models, datasets, code, and data creation pipelines as open source.

Primary area:
generative models
Keywords:
World Model, Video Generation, AI Game

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rGiCOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that generative gameplay requires reciprocity: player actions should have persistent consequences, and the generated world should act according to its own rules and state, in turn shaping the player's situation. RIGen couples three layers: (i) a state-and-rule engine that maintains persistent gameplay variables (player/enemy health, ammunition, strategy modes) and applies configurable transition rules; (ii) an interaction agent (InternVL3-8B) that infers action consequences, plans actor behavior, and reconciles predicted state with the generated video; and (iii) an audiovisual generator (LTX-2.3 22B, with FiLM-based movement/camera conditioning, button cross-attention, chunk-wise prompt routing, and causal sliding-window generation). The authors build a synchronized Xonotic dataset (95,731 recordings, 176.8 h, with audio, controls, engine state, and interaction annotations), post-train a few-step distilled generator, and demonstrate RIGen-Arena, an interactive one-on-one FPS prototype. On 100 controlled combat scenarios (50 survival / 50 death targets), RIGen reaches 81% Final-state Agreement (FSA) vs. 29% for LingBot-World 2.0 and 4% for SCOPE; fixed-generator ablations reduce FSA to 39% (motion guidance only), 35% (semantic prompting), 64% (post-hoc updates), and 76% (without reconciliation). Visual-quality metrics are largely unchanged across ablations.

Critical strengths
  1. The system decomposition and the "predict-then-reconcile" loop are a concrete answer to persistent consequences. Separating executable rules from learned generation, predicting interaction outcomes before the chunk is generated, and reconciling predicted state with committed visual evidence is a sensible architecture, and the fixed-generator ablations substantiate that each part matters: pre-generation prediction + rules + reconciliation reaches 81% FSA, while post-hoc updates reach 64% and removing reconciliation reaches 76%. This is consequential because it identifies which mechanism enables state-dependent outcomes, rather than attributing gains to a stronger generator.

  2. The outcome-control margin and the dataset are substantial contributions. 81% vs. 29%/4% FSA on prescribed combat outcomes, combined with a 176.8-hour synchronized dataset spanning local dynamics, controlled interactions, and sustained arena combat (with audio, controls, poses, and state labels), gives the community both a demonstrated capability and the data to study it. The qualitative state-dependent branches (ammo gating, low-health retreat) are convincing illustrations of gameplay that is genuinely state-driven rather than text-steered.

Critical weaknesses
  1. The "reciprocal" claim is only partially evaluated. In the quantitative benchmark, enemy motion is fixed to source annotations and all methods receive prescribed outcomes as text; this measures realization of specified outcomes, not the world acting on the player through autonomous behavior. The genuinely reciprocal part - RIGen-Arena with autonomous opponent decisions, respawns, and player defeat - is shown only qualitatively (Fig. 5). Since "reciprocal interaction" is the paper's central framing and title, the absence of any quantitative evaluation of Arena (e.g., state-transition accuracy, consistency between HUD/state and visible events, survival/death statistics, or human play ratings over multiple sessions) is a material gap.

  2. The primary metrics are LLM-judged with no human validation. FSA and Enemy Behavior are produced by Qwen3.8-27B on generated video, with unjudgeable outcomes counted as failures. This conflates generation failure with judging failure, and a 5-point difference such as 76% vs. 81% may be within judge noise. Because FSA is the main evidence for the paper's core claim, a human agreement audit on a subset (even 100 clips, reporting agreement with the judge and the frequency of unjudgeable cases) is needed. The privileged reference-guidance variant reaching 89/90/88 also suggests considerable headroom that is not discussed.

Questions for authors
  1. Can you provide any quantitative evaluation of RIGen-Arena with autonomous opponent behavior - e.g., over N sessions, the rate at which rule-engine state (health, ammo, respawns) matches visible outcomes, or human ratings of state-dependent behavior? Even a small study would directly support the reciprocity claim.

  2. What is the human-judge agreement for FSA/EB on a held-out subset, and what fraction of FSA failures are "unjudgeable" rather than visibly incorrect outcomes?

  3. Since the benchmark fixes enemy motion to source annotations, how much does the interaction agent's behavior planning contribute to FSA? Can you run the same 100 scenarios with autonomous behavior and report outcome realization?

  4. FSA-D (72%) is much lower than FSA-S (90%). What are the dominant failure modes for death targets - e.g., delayed collapse, occlusion treated as death, or respawn artifacts?

  5. The generator receives local guidance that already describes outcomes (e.g., "enemy retreats", "player is hit"). How do you ensure the generator is not merely following text, and how much of the state consistency comes from the prompt compiler rather than learned dynamics?

2 comments

We thank the reviewer for the detailed feedback on reciprocal interaction and evaluation. We respond below with additional Arena results, human evaluations, and clarifications of the benchmark setting and failure modes.

Q1. Can you quantitatively evaluate reciprocal interaction in RIGen-Arena with autonomous opponents?

We evaluate 20 continuous RIGen-Arena rollouts of 60s each with autonomous opponents and repeated encounters after respawn. We extract chunks corresponding to the events and behaviors below and use Qwen3.8-27B to check whether the generated visuals match the engine state after reconciliation or the active enemy strategy.

Event / behaviorFull RIGen consistency
Enemy death92.92% (105/113)
Player hit / injury95.09% (911/958)
Player death100.00% (58/58)
Player respawn100.00% (57/57)
Enemy behavior vs. current strategy84.20% (1263/1500)

Player HP is updated by the rule engine. The hit/injury metric checks only whether injury feedback appears in the video.

The rule engine manages ammunition and deducts one round per permitted shot, regardless of whether it hits. When ammo is depleted, firing inputs are blocked before reaching the generator.

Q2. How well do FSA/EB judgments agree with human labels, and what fraction of FSA failures are unjudgeable?

Human and VLM evaluations yield similar overall FSA/EB scores and the same method ranking, as shown in the table below. SCOPE has difficulty preserving enemy appearance after attacks, often producing blurred results that the VLM marks as unjudgeable. Human review also cannot reliably classify these cases as alive or dead, so they receive no FSA credit.

MethodEvaluatorFSAFSA-SFSA-DEBUnjudgeable / All casesUnjudgeable among FSA failures (%)
Full RIGenVLM (reported)81.0090.0072.0078.917 / 10036.84
Full RIGenHuman88.0096.0080.0080.69--
w/o reconciliationVLM (reported)76.0088.0064.0078.028 / 10033.33
w/o reconciliationHuman79.0090.0068.0079.82--
SCOPEVLM (reported)4.006.002.0035.7175 / 10078.13
SCOPEHuman4.008.000.0032.87--
LingBot-World 2.0VLM (reported)29.0058.000.0055.3211 / 10015.49
LingBot-World 2.0Human38.0076.000.0059.47--

Q3. How does autonomous planning affect outcome realization on the same 100 scenarios?

The 100 scenarios use fixed player controls and prescribed enemy motion. Once autonomous planning changes the enemy trajectory, the resulting hits and damage can change, making the original engine labels inapplicable. FSA therefore cannot be reliably evaluated against those labels.

Q4. Why is FSA-D lower than FSA-S, and what are the dominant failure modes?

  1. Distant enemies occupy a small image region and may be intermittently lost after muzzle flashes, so their death cannot be confirmed.
  2. Some scenarios contain only a limited number of shots. Hit assessment errors can prevent a successful kill within this firing budget.

Q5. Is the generator merely following outcome guidance, and how much of the state consistency comes from the prompt compiler versus learned dynamics?

This decoupling of the prompt compiler and video dynamics is intentional. The engine computes state transitions, and the prompt compiler provides outcome guidance that mainly contributes to state consistency. The generator learns to realize these outcomes under player controls. Learned dynamics also constrain the visual outcome. For example, a visibly missed shot may leave the enemy alive despite incorrect death guidance, triggering reconciliation of the engine state. Table 3 reports 89% FSA with reference guidance. With the same generator, full RIGen achieves 81% FSA versus 39% with motion guidance alone.

Q6. What explains the FSA gap between RIGen and the reference-guidance variant?

Reference guidance uses source hit annotations and rules to determine outcomes in advance, bypassing online hit assessment. In full RIGen, missed hit detections can undercount damage and prevent a kill within the limited firing budget in the original action trajectories.

Please let us know if you have any remaining questions or concerns. We would be happy to provide further clarification.

Reviewer_8W98OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RIGen, a framework for maintaining gameplay interactions in an action-conditioned video generator. It combines an explicit state and rule engine, a visual interaction agent, and an audiovisual generator. The agent predicts interaction events before generating a video chunk, and the system checks the generated result to update its retained state. The authors also collect an annotated Xonotic dataset and build an interactive Arena prototype. Experiments show improved agreement with target combat outcomes, with ablations supporting the benefit of predicting consequences before generation.

Critical strengths
  1. The method is well-motivated and the ablations are convincing for its main design choice. With the same generator, Table 3 reports 81% final-state agreement for the full method, compared with 64% for post-hoc updates and 39% for motion guidance alone. These comparisons provide useful evidence that anticipating interaction consequences helps the generator produce the intended outcome.

  2. The authors put substantial effort into the data and system implementation. The dataset covers basic dynamics, controlled interactions, and longer combat sequences, with synchronized controls and event labels. The held-out maps and enemy appearances are excluded from all task-specific training subsets. Details such as reconciling from the saved state to avoid counting damage twice also show careful consideration of how the components work together.

Critical weaknesses
  1. The quantitative evaluation is narrower than the full gameplay system presented in the paper. In the 100 benchmark scenarios, enemy motion is prescribed from annotations, and the main metric checks whether the enemy is alive or dead at the end. This does not tell us how reliably the system handles autonomous enemies, player damage, ammunition, or repeated encounters after respawn. The qualitative Arena examples are interesting, but I would like to see some quantitative evidence for this part of the system.

  2. I am not fully clear on how reconciliation handles a conflict between visible outcomes and numerical health. For example, what happens if the rules predict lethal damage but the generated enemy remains visibly alive? The paper states that observations take precedence, but the resulting health update is not sufficiently explained. In addition, final-state agreement is evaluated by a single VLM judge without human calibration. This makes the smaller improvement from 76% without reconciliation to 81% with it harder to assess.

Questions for authors
  1. Could you give concrete examples of the state updates when the enemy is visibly alive after predicted lethal damage, or visibly dead while its computed health remains positive? How often do these conflicts occur?
  2. Can you provide a focused evaluation of the Arena prototype over repeated encounters, including player damage, ammunition constraints, and respawn consistency?
2 comments

We appreciate the reviewer’s positive assessment of our design and implementation. To address the concerns, we provide additional Arena and human evaluations and explain how reconciliation handles conflicting outcomes.

Q1. How reliably does RIGen-Arena handle autonomous enemies, player damage, ammunition constraints, and repeated encounters after respawn?

We evaluate 20 continuous RIGen-Arena rollouts of 60s each with autonomous opponents and repeated encounters after respawn. Qwen3.8-27B checks whether the generated visuals agree with the engine states after reconciliation and the active enemy strategies.

Event / behaviorFull RIGen consistency
Enemy death92.92% (105/113)
Player hit / injury95.09% (911/958)
Player death100.00% (58/58)
Player respawn100.00% (57/57)
Enemy behavior vs. current strategy84.20% (1263/1500)

Player HP is updated by the rule engine. The hit/injury metric checks only whether injury feedback appears in the video.

The rule engine deducts one round per permitted shot and blocks firing when ammo is depleted. HP resets on respawn before combat resumes.

Q2. How does reconciliation resolve conflicts between numerical health and visible outcomes, and how often do these conflicts occur?

  1. If predicted damage is lethal, provisional HP becomes zero and guidance requests enemy death. After generation, if the agent verifies that the enemy remains alive, the engine restores HP to its value before the last hit before generating next chunk guidance.
  2. If predicted damage leaves positive HP, guidance specifies a surviving enemy. After generation, if the agent verifies that the enemy is dead, the engine sets HP to zero before generating next chunk guidance.

The table reports how often these two conflicts occur before reconciliation across 1500 enemy state checks from 20 RIGen-Arena 60s rollouts.

Conflict typeConflict countTotal checksConflict rate (%)
Enemy HP = 0 but visibly alive6615004.40
Enemy HP > 0 but visibly dead1015000.67

Q3. Can human evaluation validate the FSA scores and the improvement from 76% to 81% with reconciliation?

Human and VLM evaluations yield similar overall scores and the same method ranking.

MethodEvaluatorFSAFSA-SFSA-DEB
Full RIGenVLM (reported)81.0090.0072.0078.91
Full RIGenHuman88.0096.0080.0080.69
w/o reconciliationVLM (reported)76.0088.0064.0078.02
w/o reconciliationHuman79.0090.0068.0079.82
SCOPEVLM (reported)4.006.002.0035.71
SCOPEHuman4.008.000.0032.87
LingBot-World 2.0VLM (reported)29.0058.000.0055.32
LingBot-World 2.0Human38.0076.000.0059.47

Please let us know if you have any remaining questions or concerns. We would be happy to provide further clarification.

Reviewer_w2L4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RIGen is a system for interactive game/FPS generation in which player actions and the game world affect each other. A rule engine keeps game state such as health, ammo and enemy strategy. A VLM agent predicts and verifies events like hits and enemy moves. An audiovisual model renders each chunk from the player's controls and text guidance. The authors also collect a 177-hour Xonotic dataset with controls, poses and state labels. On 100 test scenarios, RIGen reaches 81% final-state agreement, against 29% for LingBot-World 2.0 and 4% for SCOPE.

Critical strengths
  • The ablations in Table 3 are clean. With the generator fixed, final-state agreement is 39 with motion guidance only and 35 when an LLM keeps the state as text, against 81 for the full system. The lesson is practical: keep game state in an executable engine and decide outcomes before generating.
  • The Xonotic dataset, with its state and outcome labels, could be useful to the community if released.
Critical weaknesses
  • The main idea is not entirely new, and much of the interaction still relies on hand-written rules. Previous work, such as Programmable World Model (Huang et al., 2026), has already explored separating game logic from visual generation. What is more interesting here is the prediction of events before generation and the correction of game states afterward, which the ablations show to be useful. However, key mechanics like health, damage, death, and strategy switching are still manually coded for a single game and opponent. It is therefore unclear how easily this approach could be applied to other games without substantial additional engineering.

  • The paper does not directly evaluate some of its main claims. The reciprocal interactions highlighted in the title are only shown in qualitative examples. The quantitative results focus on much narrower tasks: whether an enemy is dead or alive as intended, and whether its movement follows the given plan. Both are evaluated by a VLM, without human validation. The comparison with external baselines also seems somewhat unfair, since those models were not trained on the same datasets. This means the large performance gap (81 vs. 29/4) could partly come from task-specific training rather than the proposed method itself. The internal ablations provide more convincing evidence of the method's actual contribution.

Huang, Z.-H., et al. (2026). Programmable World Model. arXiv:2609.10540. https://arxiv.org/abs/2609.10540

Questions for authors
  1. How well does the VLM judge agree with human labels?
  2. Can you measure the reciprocal behavior directly? For example, do player HP and ammo, and the enemy's decisions, stay consistent with the rule state over long sessions?
  3. How often does the VLM agent mispredict or misverify events, and how do these errors accumulate?
  4. What would need to change to support a second game?
5 comments
AuthorsOpenReview ↗

Response Part 1

We thank the reviewer for the constructive feedback and for recognizing the value of our ablations and dataset. Below, we clarify our design choices and report additional results addressing the evaluation concerns.

Q1. The idea is not entirely new. How does RIGen differ from Programmable World Model?

We will discuss PWM as concurrent work in Related Work. PWM uses one-way information flow from the engine to the generator, which may leave conflicts between generated outcomes and engine state unresolved. RIGen enables bidirectional flow. Engine state guides generation, and verified video outcomes feed back to reconcile engine state.

Q2. RIGen relies on hand-written rules. How can it be adapted to other games?

Defining rules is a necessary part of game design. RIGen uses hand-written rules to make these intended mechanics explicit and executable. Adapting RIGen to another game requires changes to three layers.

  1. Fine-tune the Audiovisual Generator for the new game’s controls and interactions.
  2. Adapt the Interaction Agent to infer and verify events in the new game and map engine states and events to structured generation prompts.
  3. Configure the State and Rule Engine with the new game’s state variables and transition rules.

Q3. How well does the VLM judge agree with human labels?

Human and VLM evaluations yield similar overall scores and the same method ranking under the same settings as Table 3 in the paper. Human review resolves some cases deemed unjudgeable due to gunfire effects or off-screen enemies. VLM judgments are reliable in most other cases.

MethodEvaluatorFSAFSA-SFSA-DEB
Full RIGenVLM (reported)81.0090.0072.0078.91
Full RIGenHuman88.0096.0080.0080.69
w/o reconciliationVLM (reported)76.0088.0064.0078.02
w/o reconciliationHuman79.0090.0068.0079.82
SCOPEVLM (reported)4.006.002.0035.71
SCOPEHuman4.008.000.0032.87
LingBot-World 2.0VLM (reported)29.0058.000.0055.32
LingBot-World 2.0Human38.0076.000.0059.47
AuthorsOpenReview ↗

Response Part 2

Q4. How can reciprocal interaction be evaluated quantitatively over long sessions?

We evaluate consistency between visuals and state across 20 continuous RIGen-Arena rollouts of 60 s each, covering both player and enemy outcomes and state-dependent enemy behavior. We extract the corresponding video chunks and use Qwen3.8-27B to check whether the generated visuals agree with the engine state after reconciliation or the enemy strategy active in each chunk.

Event / behaviorFull RIGen consistency
Enemy death92.92% (105/113)
Player hit / injury95.09% (911/958)
Player death100.00% (58/58)
Player respawn100.00% (57/57)
Enemy behavior vs. current strategy84.20% (1263/1500)

Player HP is updated by the rule engine. The hit/injury metric checks only whether injury feedback appears in the video.

HP and ammunition are maintained by the rule engine. Hits apply configured damage, each permitted shot consumes one round, and firing is blocked when ammo is depleted.

Q5. How often does the VLM agent mispredict or misverify events, and how do these errors accumulate?

We use Qwen3.8-27B as an external visual judge to audit the agent’s outputs before and after generation across 20 RIGen-Arena rollouts of 60 s each.

Event prediction and behavior assessment (before generation)

Agent functionMeasureMismatch countTotal checksMismatch rate (%)
Shot predictionMismatch between prediction and realization12216487.40
Enemy attack preconditionsVisibility and facing assessment error6815004.53
Movement feasibilityIncorrect feasibility judgment615000.40

Event verification (after generation)

Agent functionMeasureError countTotal checksError rate (%)
Shot verificationHit/miss classification error2516481.52
Enemy state verificationAlive/dead/not visible classification error1115000.73
Player death verificationDeath-state classification error0580.00
Player respawn verificationRespawn-event classification error0570.00

We observed no severe error accumulation in the 20 evaluated sessions. Errors mainly shifted enemy kill timing or player survival time. Player and enemy HP reset on their respective respawns.

Q6. The baselines were not trained on the same dataset. Is the comparison fair?

Both baselines support interactive FPS gameplay, yet almost never realize the intended enemy deaths in our benchmark (FSA-D is 2% for SCOPE and 0% for LingBot-World 2.0), even with explicit outcome guidance. Enabling these interactions is a central contribution of RIGen, including our dataset and training. With the same trained generator, removing interaction guidance reduces FSA from 81% to 39% (Table 3). This gap reflects our framework’s contribution, not differences in training data.

Please let us know if you have any remaining questions or concerns. We would be happy to provide further clarification.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Revised manuscript uploaded

We thank the reviewers and the Area Chair for their help. We have uploaded an updated PDF with additional human evaluation, continuous Arena evaluation, and a discussion of PWM. These additions are highlighted in blue.

AuthorsOpenReview ↗

Supplemental materials updated

Dear reviewers and Area Chair,

We have added evaluation protocol details to the appendix of the revised PDF. When you have a chance, we would appreciate your thoughts on our responses and would be happy to clarify any remaining questions.

Area_Chair_kepyOpenReview ↗

Discussion phase will end on Nov 18

Dear Reviewers,

This is a kind reminder that discussion phase will end on Nov. 18. As the authors have provided the rebuttal, could you please check authors' rebuttal and update your thoughts? Your early feedback would make the discussion more smooth and informative.

Thank you.

AC

AuthorsOpenReview ↗

Summary of Reviewer Concerns and Revisions

Dear Area Chair and Reviewers,

We thank the Area Chair for guiding the discussion and the reviewers for their time.

RIGen shows how a video generator can power a game in which the player’s actions change the world and the world’s own actions change the player’s state. The four points below summarize the main concerns raised across the three reviews and how we addressed them.

  1. How does RIGen differ from the concurrent Programmable World Model (PWM)? RIGen operates without the explicit 3D engine used by PWM and uses verified outcomes from the generated video to correct its stored game state.
  2. Are the VLM scores reliable? Human and VLM scores were similar overall. Human review confirmed both the ranking of all four methods and the benefit of reconciliation (Table 4 and Appendix C.5).
  3. Can reciprocal interaction be measured directly? We added a quantitative evaluation of how enemy actions affect the player.
  4. How accurate and stable is RIGen-Arena during continuous play? We evaluated the system over 20 one-minute sessions with autonomous opponents and repeated encounters after respawn (Table 5 and Appendix C.6).

We address the remaining questions in our individual replies.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.