acceptodds
Under review as a conference paper at ICLR 2027

H3-World: Turning Language Understanding into World Control

Abstract

We present H3-World, an efficient framework that turns the 33B MiniMax-H3 video generator into an interactive world model. Our key finding is that, as large video generators become more capable, language is emerging as a natural interface for control. MiniMax-H3, for example, already supports zero-shot control of character behavior and camera motion through natural-language instructions. Building on this, H3-World turns this coarse language interface into precise, temporally grounded world control, without introducing dedicated action modules. Specifically, we represent each action as a structured combination of character and camera instructions, and align them with the corresponding temporal video latents. To make the control temporally precise, we further introduce temporal attention routing, which restricts each instruction to its intended time interval and reduces control leakage across actions. Importantly, H3-World directly reuses the semantic representations learned during large-scale video pretraining and requires only lightweight adaptation. With only 8,000 gameplay samples, 10,000 LoRA optimization steps, and 0.199% trainable parameters, H3-World achieves effective character and camera control while preserving strong generation quality. It also generalizes to unseen scenarios. These results show that the control capabilities emerging in large video generators can be efficiently transformed into interactive world control. Anonymous project page is available at https://anonymous.4open.science/w/H3-WORLD-website-6C01/.

Primary area:
generative models
Keywords:
World Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_38t6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents H3-WORLD, which aims to transform the coarse-grained understanding of motion already present in pretrained video generation models into temporally grounded action control. The method converts character and camera controls into compositional natural-language instructions, binds each instruction to its corresponding video latent interval, and uses single-egress attention routing to restrict the direct injection of action information into unmatched temporal intervals. Adaptation is performed using LoRA while keeping the MiniMax-H3 backbone frozen.

Critical strengths
  1. I agree with the paper’s premise: powerful pretrained video models can already generate approximately correct motion from text, and developing fine-grained control interfaces should build on these existing capabilities as much as possible. Converting discrete controls into structured natural language and integrating them into the model’s native text pathway through lightweight adaptation is a reasonable technical approach. By training only 65.6M parameters, the method achieves effective character and camera control, supporting the feasibility of parameter-efficient adaptation for this task.

  2. Expressing actions in natural language alone does not guarantee that the model will execute them at the correct time. The paper further introduces per-latent temporal binding and single-egress routing to constrain the injection of action information within H3’s shared bidirectional attention.

Critical weaknesses
  1. In the rightward movement example in the second row of the demo videos, the dominant motion I observed appears to be forward movement or movement along the slope, with clear lateral movement to the right appearing only in the later part of the video. This leaves me uncertain whether the rightward movement command is executed promptly and continuously within the specified interval, rather than merely producing related motion near the end. In another pitch-down example, the character also appears to move forward. If the input does not request character movement, but instead asks the character to remain in place while only the camera tilts downward, this unintended character motion suggests interference between character and camera control.

  2. This also leads to my main concern: separating character and camera actions in the textual representation does not automatically guarantee independent control of the generated behavior; single-egress routing primarily constrains direct information injection across temporal intervals and does not directly guarantee independence between different control dimensions.

Questions for authors

My questions mainly concern the demonstrated results and whether the method actually works. Please refer to the Weaknesses section for details.

3 comments

We sincerely thank Reviewer 38t6 for carefully examining our demonstrations and recognizing the value of building action control on pretrained video models. We address the concerns below with further analysis and quantitative evaluations.

>>> W1 "Move right" and "tilt down" demos

>>> A1 Thank you for the careful observations. We acknowledge that unintended motion can occur in some cases, including the two highlighted demos. We identify the following possible causes.

(1) "Move right" demo.

  • Downhill motion. The model may reproduce downhill movement patterns learned from its pretrained prior or similar scenes in training. On flat ground, the character moves clearly sideways under right-strafe commands.
  • Training data. Rightward movement often occurs together with forward or backward movement in ABot. Only 19.8% of right-strafe training latents contain pure sideways movement.

(2) "Tilt down" demo.

  • Limited tilt data. Tilt is under-represented in training, accounting for only 23.7% of camera rotation, with a median rotation of 3.9° per key press compared with 9.7° for yaw.
  • Unintended character movement. Character movement can also appear under tilt-only commands in recorded gameplay. Our quantitative evaluation in A2 shows that H3-World exhibits a similar rate.

These observations help explain the occasional control imperfections. We further examine action timing and control independence in A2.

>>> W2 Independent and timely action control

>>> A2 Thank you for this insightful point. Indeed, our method may not guarantee perfect independence between camera control and character control. However, we've made tremendous efforts in the method design to fulfill the disentanglement between different control types.

For example, we design factorized text clauses to represent character and camera actions separately, and temporal binding and single-egress routing to ensure that control signals are precisely aligned with their corresponding video latents.

We further evaluate these capabilities through four tests.

(1) Unintended character movement. We measure character movement under camera-only commands, using recorded gameplay as the reference, since camera rotation can also cause apparent character motion.

Character movement detected (IDM)Recorded gameplayH3-World
Pan-only commands25.6%27.8%
Tilt-only commands11.7%12.4%

H3-World produces only 2.2 and 0.7 percentage points more detected character movement than recorded gameplay, respectively, indicating limited additional interference between character and camera controls.

(2) Single-action control. We evaluate 68 single-action cases from WBench, excluding 5 cases with combined actions.

MethodNavigation score ↑Translation error ↓Rotation error ↓
Frozen H30.5920.4230.394
H3-World0.6960.3720.236

H3-World improves the navigation score by +0.105 (95% CI [0.042, 0.167]) and reduces rotation error by 0.158, demonstrating more accurate control when only one action is requested.

(3) Action composition. We evaluate unseen combinations by adding tilt to seen movement + pan commands.

SettingMovement F1 ↑Yaw F1 ↑Pitch F1 ↑
Seen: movement + pan0.3490.745–
Seen: pan + tilt–0.8030.265
Unseen: movement + pan + tilt0.3870.7210.135

Movement and yaw remain comparable to their seen components, showing that the model can combine these controls in unseen combinations, although pitch remains weaker.

(4) Timely action execution. H3-World follows action schedules that change within a clip, including a mid-clip camera reversal that frozen H3 cannot follow (Section 4.3). Removing temporal binding or routing reduces yaw and pitch F1 by 47–61% (Section 4.5).

These results demonstrate that our model executes temporally specified actions, rather than merely producing related motion near the end of a video.

We add the unintended-movement analysis to Appendix B.7 and the composition results to Section 4.6, alongside the single-action WBench evaluation.

>>> Q Does the method work?

>>> A3 Of course, our method works. H3-World effectively controls character and camera motion, with clear improvements over frozen H3 in controllability, temporal grounding, and generalization.

(1) Effective action control. We compare frozen H3 and H3-World on the same subset of held-out clips.

MethodMovement F1 ↑Yaw F1 ↑Pitch F1 ↑Macro F1 ↑
Frozen H30.5070.6110.1170.449
H3-World0.5100.8320.4170.588

H3-World improves yaw F1 by 0.221, pitch F1 by 0.300 (3.6×), and overall macro F1 from 0.449 to 0.588. These gains demonstrate that our adaptation substantially improves action controllability beyond H3's pretrained prior.

(2) Effective temporal control. As demonstrated in A2, H3-World follows changing action schedules that frozen H3 cannot. The temporal-binding and routing ablations further confirm that our method controls when actions occur, not just what motion is generated (Sections 4.3 and 4.5).

(3) Generalization. As shown in A2, movement and yaw remain controllable in unseen action combinations. On 73 unseen WBench cases, H3-World also improves the navigation score from 0.580 to 0.680 (Section 4.6).

Together, these results demonstrate that H3-World delivers effective, temporally grounded action control, and that the issues observed in individual demos do not represent its overall performance.

Dear Reviewer 38t6,

As the discussion period is approaching its end, we would like to kindly follow up on our rebuttal and ask whether you have any remaining questions or concerns.

We would be happy to provide further clarification if needed.

Thank you very much for your time and consideration!

Best regards,
The Authors

Reviewer_rvXTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents H3-WORLD, a framework that converts the pretrained 33B-parameter MiniMax-H3 audio-video generator into an interactive, action-controllable world model without adding a dedicated action-conditioning module. The central thesis is that large video generators already contain a coarse language-based control prior, so control can be expressed in the model's native text space rather than learned from scratch.

Critical strengths
  1. Striking parameter/data efficiency with a clean causal story for where control comes from. H3-WORLD matches or exceeds a geometry-supervised baseline (SolarWM) on 3 of 4 control channels using 3.2% of its trainable parameters and no camera-pose supervision, from only 8,000 clips. This is consequential because it is not merely a good number — the Section 4.3 analysis demonstrates why it works (reusing a pretrained motion prior and converting it to temporally grounded control via lightweight adaptation), which makes the result transferable rather than a one-off system. This is the single consideration that most moves me toward acceptance.

  2. A simple, parameter-free mechanism (single-egress routing) that is validated, not just proposed. The routing + temporal-binding design directly targets the real failure mode of bidirectional single-stream attention (control leakage across time), and the ablations show both components are necessary, with an interpretable asymmetry between camera and locomotion sensitivity (yaw/pitch degrade 47–61% vs. movement 14–18% when grounding is removed). The design is well-matched to the architecture it operates in.

Critical weaknesses
  1. The headline "preserves generation quality" claim is essentially unquantified. The paper's thesis is that you can gain control without the degradation that heavy adaptation causes — yet generation quality is shown only through curated qualitative figures, with no FVD/quality metric and no long-rollout stability analysis. Because this is a load-bearing claim (it is the stated advantage over "extensive adaptation [that] may disturb pretraining capabilities"), its lack of quantitative support is the weakness that most limits the paper's current standing.

  2. Narrow evaluation scope relative to the generality of the claims. The paper argues broadly that language can serve as a practical control interface for turning pretrained video generators into world models, but validates on one dataset, one backbone, and short fixed-length rollouts, with generalization shown qualitatively. The gap between the breadth of the thesis and the breadth of the evidence is the main reason I cannot rate this higher than a weak accept. The authors acknowledge the short/fixed-horizon limitation, but longer-horizon, persistence, and real-time interaction are exactly where this subfield's hard problems lie.

Questions for authors
  1. Can you provide a quantitative generation-quality comparison (e.g., FVD or a perceptual metric) between frozen MiniMax-H3 and H3-WORLD on the held-out set, and ideally quality/consistency as a function of rollout length? This directly bears on the "preserves generation quality" claim.
  2. SolarWM outperforms H3-WORLD on pitch (0.481 vs. 0.416). Do you have an explanation — e.g., is pitch underrepresented in the textual action vocabulary or the training distribution, and would it respond to tilt-specific supervision?
  3. The method's premise is that it reuses MiniMax-H3's pre-existing language control prior. Have you tried the same recipe on a backbone with a weaker native motion-language prior? Evidence that the gains are not almost entirely inherited from this particular backbone would substantially strengthen the generality claim.
3 comments

We sincerely thank reviewer rvXT for highlighting the parameter and data efficiency of our method and the validated routing design.

>>> W1 & Q1 Generation quality

>>> A1 Thank you for this important question. As suggested, we compare generation quality with frozen H3 on the held-out set and test longer rollouts. H3-World maintains comparable visual quality and achieves better FVD and FID.

(1) Quality against frozen H3. We compare H3-World with frozen H3 on 150 held-out clips, using the same first frames and 3 seeds. Frozen H3 receives the action schedule as one global prompt.

Following WBench, Aesthetic and Imaging measure visual and technical quality, Background measures frame-to-frame consistency, and Dynamic measures the share of videos with clear motion. FVD and FID measure the distance to recorded gameplay.

MethodAesthetic ↑Imaging ↑Background ↑Dynamic ↑FVD ↓FID ↓
Recorded gameplay0.5650.6560.9550.99––
Frozen H30.5740.6600.9400.92140.646.8
H3-World0.5730.6440.9490.98120.341.4

H3-World lowers FVD and FID and improves Background and Dynamic. Aesthetic is unchanged, and Imaging differs by only 0.016. H3-World therefore preserves the generation quality of frozen H3. We add this comparison to Section 4.4 (Table 3) of the revised paper.

(2) Longer rollouts. A systematic long-horizon study requires a causal model, which we leave to future work. Without retraining, we test chunk-by-chunk generation, where seven 22-frame chunks (6.2 s) connect without visible seams, preserve the scene and character, and follow the commanded keys.

(3) Pretrained capabilities. H3-World trains only a 65.6M-parameter LoRA (0.2% of H3) on 8K clips, keeping all base weights frozen. This avoids full fine-tuning on large single-game datasets that tie a model to one domain. The unseen-scene results in A2 further confirm that H3-World retains H3's generality.

>>> W2 Evaluation scope

>>> A2 Thank you for this comment. We agree that the evidence should match the breadth of the claims. We now quantify generalization and clarify the scope.

(1) Generalization.

  • Unseen scenes. We evaluate on 73 third-person navigation cases from WBench, outside our training data. The navigation score compares the camera trajectory recovered by MegaSaM with the commanded motion.
Unseen scenes (WBench)Navigation score ↑Translation error ↓Rotation error ↓
Frozen H30.5800.4230.417
H3-World0.6800.3780.262

H3-World improves the navigation score by +0.10 (paired 95% CI [0.04, 0.16]), mainly through a lower rotation error.

  • Unseen action combinations. Adding tilt to seen movement + pan commands maintains comparable movement and yaw F1 to the seen components (0.387/0.721 vs. 0.349/0.745).

We add both tests to Section 4.6 of the revised paper.

(2) Dataset. Although we use only one dataset, ABot itself covers multiple games and scenes. Moreover, our method is not limited to ABot and can be applied to other datasets with action–frame paired video data, as a rule table converts actions into text clauses.

(3) Backbone and rollout length. We discuss a weaker backbone in A4 and longer rollouts in A1.

Long-horizon persistence and real-time interaction require causal generation, typically obtained by distilling a bidirectional teacher into a causal student. Such a student inherits the teacher's control, making temporally grounded control in a bidirectional world model the foundation for these goals.

H3-World provides this foundation, and the chunk-by-chunk prototype in A1 demonstrates longer rollouts without retraining.

>>> Q2 Pitch vs. SolarWM

>>> A3 Thank you for this question. We attribute the weaker pitch performance to two main factors:

  1. Weak pretrained prior. Frozen H3 responds weakly to tilt, with a pitch F1 of only 0.117 vs. 0.611 for yaw (A4).
  2. Under-represented pitch data. Pitch accounts for only 23.7% of camera rotation (yaw: 76.3%), and each tilt key press produces a smaller rotation (3.9° vs. 9.7° for yaw). We add these statistics to Appendix A.2.

In contrast, SolarWM uses per-frame camera-pose supervision, which directly encodes pitch and achieves higher F1 (0.481 vs. 0.416), but requires 32× more trainable parameters (2,075M vs. 65.6M) and per-frame pose annotations.

>>> Q3 Weaker backbone

>>> A4 Thank you for this question. Actually, a recent work, Incantation [1] demonstrates language-based action control on Wan2.2-5B, showing that this approach is not limited to H3. However, the two approaches differ in their architectures and reliance on pretrained priors.

(1) Applicability to other backbones. Incantation uses per-timestep language actions with Wan2.2-5B, but requires substantial scene-specific fine-tuning data and shows limited generalization beyond its training game. In contrast, H3-World leverages H3's strong pretrained language-motion prior, enabling effective control with only 8K clips and a small LoRA.

Our temporal binding and single-egress routing are specifically designed for H3's joint self-attention. Since Wan2.2 uses text cross-attention, the same recipe cannot be directly transferred without an architecture-specific redesign.

(2) Contribution beyond the pretrained prior. To examine whether our gains are merely inherited from H3, we compare frozen H3 and H3-World on the same subset of held-out clips.

MethodMovementYawPitchAll
Frozen H3 (prior only)0.5070.6110.1170.449
H3-World (prior + recipe)0.5100.8320.4170.588

While movement stays similar, yaw improves by 0.22 and pitch improves by 3.6× (0.117 → 0.417).Moreover, frozen H3 cannot follow the timing of action commands (Section 4.3), while removing routing or temporal binding reduces yaw and pitch F1 by 47–61% (Section 4.5).

These results show that the pretrained prior provides a foundation, but our adaptation is necessary to convert it into temporally grounded action control.

Reference

[1] Zhu, Shangwen, et al. "Incantation: Natural language as the action interface for multi-entity video world models." arXiv preprint arXiv:2605.18601 (2026).

The authors have answered all of my questions, but the overall quality of this paper does not justify a score of 4, so I have decided to maintain my original score.

Reviewer_w9FNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

H3-World turns the 33B MiniMax-H3 video generator into an action-conditioned world model. Keyboard and camera states become short text clauses, one per video latent. Each clause is tied to its latent by a temporal position offset and a “single-egress” attention mask, and LoRA (65.6M parameters, 0.199%) is trained on 8,000 gameplay clips. On 150 held-out clips, controllability is measured by an inverse-dynamics model (F1 per control group). H3-World beats additive-embedding, FiLM and SolarWM baselines on movement, yaw and speed, but not on pitch. Ablations show that removing routing or temporal binding hurts camera control. Generalization to unseen action pairs and new scenes is shown qualitatively.

Critical strengths

The method is simple and cheap. I do love the simplicity. It needs no new action module, a small LoRA, and 8K clips, and the temporal binding and routing are described precisely enough to reimplement. This makes it a practical recipe for adapting a large video model.

The evaluation setup is better than most in this area. It uses a ceiling and floor, an IDM with reported accuracy, and a flow-based test of whether a camera command happens at the right time (Table 6). The ablation cleanly shows that temporal binding and routing matter for camera control.

Critical weaknesses
  1. The headline claim, that the pretrained language prior is what makes the interface work, is not isolated. There is no baseline with learned (non-text) action tokens using the same binding, routing and LoRA. Also, not sure if i feel very surprised to read that "language is emerging as a natural interface for control", isn't it something everyone knows?

  2. Several key claims have no numbers. “Preserving generation quality” has no quality metric against frozen H3. Generalization to the 52 unseen combinations and to new scenes is only shown with hand-picked figures, with no seen vs unseen comparison.

  3. The “interactive world model” claim is overstated. The model denoises a whole 124-frame future given the full scheduled action sequence, so it is neither causal nor closed-loop, and the action space is tiny (9 character clauses x 16 camera clauses).

Questions for authors
  1. Can you add a learned-action-token variant with the same binding, routing and LoRA, and a comparison to Incantation? This would tell me whether text matters at all.

  2. Can you report video-quality metrics against frozen H3, and seen vs unseen combination controllability with confidence intervals or multiple seeds?

  3. Does the method work on a second backbone? (A suggestion, not a condition)

5 comments

We sincerely thank reviewer w9FN for appreciating the simplicity of our method and the rigor of our evaluation setup. We have answered the questions below and revised the paper accordingly.

>>> W1 & Q1 Role of the language prior

>>> A1 Thank you for this suggestion. As suggested, we add a learned-action-token variant with the same binding, routing and LoRA, and a comparison with Incantation [1].

(1) Learned action tokens.

The variant replaces each character or camera clause with 6 trainable action tokens and keeps everything else of H3-World unchanged. We compare it with H3-World and frozen H3 on a subset of the held-out clips.

MethodMovementYawPitchAll
Frozen H3 (global prompt)0.5070.6110.1170.449
Learned action tokens0.3610.3010.1290.301
H3-World (language)0.5100.8320.4170.588

The learned tokens reach an IDM macro F1 of only 0.301, below H3-World (0.588) and even below frozen H3 (0.449). Without the language prior, the LoRA must learn each action from scratch, which 8K clips do not support. The text is therefore what makes lightweight adaptation sufficient. We add this comparison to Section 4.2 (Table 2) of the revised paper.

(2) Comparison with Incantation [1].

Incantation also expresses per-timestep actions in language, but it overfits to Elden Ring, while H3-World generalizes.

  • Results. On our OOD first frames, Incantation turns the scene into Elden Ring within about 1 s. Conversely, Elden Ring lies outside our training data. On its first frames, H3-World keeps the scene and the character, and follows the movement and camera commands.
  • Design. The two differ in backbone (Wan2.2-5B vs. 33B H3), conditioning (cross-attention vs. routed joint self-attention) and training (single-game fine-tuning vs. a LoRA on a subset of the multi-game ABot dataset).

We add this comparison to Appendix C.3 of the revised paper.

(3) "Language as a natural interface".

We agree that language as a control interface is not a new concept. We demonstrate how MiniMax-H3's pretrained language prior can be leveraged for action control in video generation.

H3's single-stream architecture requires a distinct adaptation, motivating our design of temporal binding and single-egress routing.

To isolate the role of the language prior, we compare against learned action tokens under the same binding, routing, and LoRA setup. The results show that language conditioning enables effective control with only 8K clips, whereas learned tokens perform worse.

In contrast, applying a similar language-based interface to Wan2.2-5B requires substantially more data from the same scene to achieve effective control.

>>> W2 & Q2 Quality and generalization

>>> A2 Thank you for these suggestions. To address the concerns about generation quality and generalization, we provide video-quality metrics against frozen H3, quantify controllability on seen vs. unseen action combinations with 95% confidence intervals, and further evaluate generalization to unseen scenes.

(1) Generation quality. We compare H3-World with frozen H3 on 150 held-out clips, using the same first frames and 3 seeds. We use the WBench metrics: Aesthetic and Imaging rate visual and technical quality, Background measures frame-to-frame consistency, and Dynamic is the share of videos with clear motion. FVD and FID measure the distance to recorded gameplay.

MethodAesthetic ↑Imaging ↑Background ↑Dynamic ↑FVD ↓FID ↓
Recorded gameplay0.5650.6560.9550.99––
Frozen H30.5740.6600.9400.92140.646.8
H3-World0.5730.6440.9490.98120.341.4

H3-World maintains comparable visual quality to frozen H3, with improved Background consistency and Dynamic. It also achieves lower FVD and FID, indicating closer alignment with recorded gameplay. We add this comparison to Section 4.4 (Table 3) of the revised paper.

(2) Seen vs. unseen combinations. We select 4 representative combinations from the 52 unseen ones. We compare them with their seen action components using 288 videos generated from 24 held-out first frames. We report 95% bootstrap confidence intervals computed over first frames.

SettingMovement ↑Yaw ↑Pitch ↑
Seen: movement + pan0.349 [0.30, 0.39]0.745 [0.72, 0.77]–
Seen: pan + tilt–0.803 [0.77, 0.83]0.265 [0.23, 0.30]
Unseen: movement + pan + tilt0.387 [0.34, 0.43]0.721 [0.69, 0.75]0.135 [0.10, 0.17]

For unseen combinations, movement and yaw perform comparably to their seen components, demonstrating that character and camera controls can generalize to new combinations. Pitch performs worse, likely because tilt rarely co-occurs with movement during training (only 0.1% of tilt latents). We include these results in Section 4.6 (Table 5) and Appendix B.6 of the revised paper.

(3) Unseen scenes. We further use the 73 third-person navigation cases of WBench, which lie outside our training data. Its navigation score compares the camera trajectory recovered by MegaSaM with the commanded motion.

Unseen scenes (WBench)Navigation score ↑Translation error ↓Rotation error ↓
Frozen H30.5800.4230.417
H3-World0.6800.3780.262

H3-World raises the navigation score by +0.10 (paired 95% CI [0.04, 0.16]), mainly through a lower rotation error. We add this test to Section 4.6 (Table 6) of the revised paper.

>>> W3 "Interactive" claim and action space

>>> A3 Thank you for pointing this out. We acknowledge that the term is somewhat overstated. We use it in the sense of per-latent action control, where latents follow their own commands, and we now add this qualifier to the main text.

(1) Causal, closed-loop rollout is indeed central to interactive world models. Within the time, compute and data of the rebuttal period, we implement chunk-by-chunk generation with our model, without retraining, to examine both properties.

  • Closed loop. Each 22-frame chunk starts from the last frame of the previous chunk with a newly issued action. Seven chunks (6.2 s) join without visible seams and largely follow the commanded keys (IDM recall 0.71 vs. 0.66 for one 124-frame clip).
  • Causal, toward real time. Each chunk depends only on the past, and an 8-step Turbo LoRA reduces denoising from 38 s to 7.5 s per chunk.

We add this prototype to Appendix D of the revised paper.

(2) Action space. The 9 × 16 action space follows the game controls recorded in ABot, similar to WBench's discrete navigation protocol. A rule table maps the key states recorded in ABot (WASD for movement, IJKL for camera, and F for speed) into text clauses. The interface is easily extensible: new actions, such as jumping or attacking, can be added through new clauses without changing the architecture.

>>> Q3 Second backbone

>>> A4 Thank you for the suggestion. Language-based action control is applicable to other backbones, but the adaptation depends on their architectures.

  • Applicability to other backbones. As shown in A1, Incantation [1] demonstrates language-based action control on Wan2.2-5B, but requires substantial scene-specific training data and shows limited generalization. In contrast, H3-World leverages H3's pretrained prior to achieve effective control with only 8K clips and a small LoRA.
  • Architecture-specific adaptation. Our temporal binding and single-egress routing are designed for H3's single-stream architecture. Wan2.2 uses text cross-attention, so our routing cannot be directly transferred without an architecture-specific redesign.

Reference

[1] Zhu, Shangwen, et al. "Incantation: Natural language as the action interface for multi-entity video world models." arXiv preprint arXiv:2605.18601 (2026).

Comments on the paper

2 comments

We sincerely thank all reviewers for their constructive feedback. We deeply appreciate the following positive comments:

  • The method is simple and parameter-efficient: Reviewer w9FN, Reviewer rvXT, Reviewer 38t6
  • Building on the native language pathway is reasonable: Reviewer rvXT, Reviewer 38t6
  • Temporal binding and routing suit the architecture and are validated by ablations: Reviewer w9FN, Reviewer rvXT, Reviewer 38t6
  • The evaluation is rigorous: Reviewer w9FN

We will address the specific questions and concerns raised by the reviewers in the subsequent sections of this rebuttal.

AuthorsOpenReview ↗

Summary of Discussion and Rebuttal

Dear Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

We sincerely thank all reviewers for their valuable feedback. We would like to briefly summarize the recognized strengths and substantial improvements made during the rebuttal.

1. Strengths Recognized by Reviewers

  • Parameter and data efficiency: Only 65.6M trainable parameters (0.199%) and 8K clips. w9FN, rvXT, 38t6
  • Simple and effective design: Leveraging pretrained language priors with well-motivated temporal binding and routing. w9FN, rvXT, 38t6
  • Rigorous evaluation: Clear ablations and well-designed controllability metrics. w9FN

2. Major Additions During Rebuttal

We conducted substantial new experiments to address the reviewers' concerns:

  • Language prior (w9FN): H3-World outperforms learned action tokens (0.588 vs. 0.301 macro F1), validating our core motivation.
  • Generation quality (w9FN, rvXT): Maintains comparable visual quality while improving FVD (120.3 vs. 140.6) and FID (41.4 vs. 46.8) over frozen H3.
  • Generalization (w9FN, rvXT, 38t6): Improves navigation scores on 73 unseen WBench cases (0.680 vs. 0.580), with additional quantitative evaluation of unseen action combinations.
  • Control independence (38t6): Camera-only commands introduce only 0.7–2.2 percentage points of additional detected character movement compared with recorded gameplay.
  • Longer rollouts (w9FN, rvXT): Demonstrated chunk-by-chunk generation without retraining, extending beyond the original fixed-length generation.
  • Backbone applicability (w9FN, rvXT): Added comparison with Incantation and clarified architectural limitations.

3. Summary

Overall, the rebuttal substantially strengthens the original submission, with additional quantitative evidence addressing the major concerns. We believe these results further demonstrate the methodological contributions, parameter efficiency, and effectiveness of H3-World.

In addition, we have addressed the specific concerns raised by each reviewer and provided detailed responses in the comment sections. We have also incorporated the necessary changes into the revised manuscript.

Finally, we sincerely appreciate the reviewers' valuable feedback and thank all reviewers, ACs, SACs, and PCs for their time and dedication to the review process.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.