acceptodds
Withdrawn from ICLR 2027

HelloWorld: Enabling Socially Interactive Characters in Video World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Despite the remarkable recent progress of video world models, social interaction between users and the characters within these worlds remains unsupported. To fill this gap, we present HelloWorld, a video world model that enables social interaction with in-world characters. With a single button press, users can prompt the on-screen character to respond toward the camera, e.g., turning to the viewer, waving, nodding, or speaking a short greeting. To make these interactions natural, we propose a self-distillation pipeline that finetunes the video generation model on data synthesized by itself. Each synthesized clip contains both social interactions and camera motion, allowing the model to learn camera-pose conditioning without degrading interaction quality. At inference, we further introduce a training-free module that determines when the interaction occurs. Upon a button press, it modulates the cross-attention masks of the DiT so that the interaction-related text prompt attends only to the frames within the press window, temporally localizing the character's response. We further build HelloWorldBench, a 400-sample benchmark with three social interaction metrics alongside three conventional metrics, for evaluation. Experiments demonstrate that HelloWorld surpasses a variety of baselines in interaction quality, while maintaining state-of-the-art picture aesthetics and camera-pose following.

Primary area:
generative models
Keywords:
Video World Models, Multimodal Social Interactions, Interactive Video Generation Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_tcGzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HelloWorld, which converts the audio-video generator LTX-2.3 into a camera-controllable model whose on-screen characters can perform viewer-directed social actions (waving, greeting, speaking) within a user-specified time window.

The method has two parts:

  • Training: a LoRA finetuned with warp-video conditioning (following Warp-as-History). The training data are 156 clips generated by the base model itself, with camera poses and first-frame point clouds recovered by Pi3X. The camera prompt is removed so that camera control comes from the warp video.
  • Inference: a training-free temporal cross-attention mask (Eq. 4) that blocks queries outside the press window from attending to interaction-prompt tokens.

The authors also introduce HelloWorldBench: 400 samples over 120 images, with VLM-judged ActAcc and TimeAcc, a gaze-based GazeDev, and standard quality and camera metrics. Validation covers a comparison with six baselines (Table 1), ablations on training data and masks (Tables 2 and 3), a cost analysis (Table 4), and a 30-rater user study (Table 5).

Critical strengths
  1. A new, practical problem with a simple recipe that preserves interaction ability. Viewer-directed social interaction is a reasonable new axis for controllable video generation. The self-distillation pipeline needs no external data or annotation. It largely retains the base model's interaction ability during camera finetuning, whereas real-video finetuning degrades it (Table 2). The user study shows a clear interaction preference over the real-video LoRA (Table 5), a larger gap than the automatic metrics suggest. This is the main evidence that the choice of training data matters.

  2. Simple temporal control and an initial benchmark. Within HelloWorld, the training-free mask raises TimeAcc from 36.7 to 81.7 and SpeechInWin from 52.5 to 69.1 (Table 3), although TimeAcc is a coarse three-bin VLM judgment. HelloWorldBench decomposes interaction into what is performed, when, and whether it is directed toward the viewer (Sec. 4.2). This makes it a useful starting point for the task, pending validation of its metrics.

Critical weaknesses
  1. Modest technical novelty, and a framing broader than what the system supports.

    a. Each component builds on existing ideas. Warp-video conditioning follows Wang & He (2026), the training data come from the base model itself, and the temporal mask is a simple restriction of text cross-attention. The contribution lies mainly in combining these pieces for a new task.

    b. The "single button press" framing (Abstract, Fig. 1) is broader than what the system supports. HelloWorld denoises the whole 10 s clip jointly, so the camera trajectory and interaction window must be fixed before generation and cannot respond to inputs given during it, regardless of speed, as the authors acknowledge (Sec. 6). A more accurate description is camera-controlled image-to-video generation with timed interaction prompts.

  2. TimeAcc, the metric behind the largest reported gain, is coarse and unvalidated.

    a. Granularity. The VLM chooses among three uniform segments (Sec. 4.2), and the assigned windows are exactly early, middle or late (Sec. 4.1). TimeAcc therefore only checks whether the interaction falls in the correct third of the clip, a resolution of roughly 3.3 s. Because the judge must pick a single segment, the metric cannot measure onset error or detect interactions that also occur outside the window. This is too coarse to support the claim that the response occurs "precisely within the press window" (Sec. 3.3).

    b. Validity. Temporal localization is demanding for VLMs, yet the paper does not report how well the VLM judge agrees with human judgments. Videos judged as "no interaction" are excluded without reporting how many, which can inflate TimeAcc for methods that often fail to interact.

    This matters because TimeAcc carries the largest margin in Table 1 and is the main evidence for the temporal mask (Table 3). A finer and validated measure, such as human-annotated onset error in seconds on a subset, would make this claim much more convincing.

Questions for authors
  1. TimeAcc only checks whether the interaction falls in the correct third of the clip (Sec. 4.2). Could you report the start and end times of each interaction relative to the press window, ideally human-annotated on a subset? This would show whether interactions begin before the press or extend beyond the window.
  2. Have the VLM judgments for ActAcc and TimeAcc been validated against human labels?
  3. Neither Sec. 5.4 nor Appendix A reports how much real data the Real-video LoRA used. If it differs substantially from the 156 self-generated clips, Table 2 and the user study (Table 5) confound data content with data quantity, which weakens the main evidence for self-distillation.
Reviewer_gSw8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method for enabling the controlled triggering of social interactions in world models.

To achieve this, the authors extend a prior approach, Warp-as-History (Wang and He, 2026), which transforms a video generation model into a world model. Specifically, they apply this approach to videos generated by the video model itself with specified social interactions. At inference time, they introduce an attention mask on the interaction condition to restrict its attention to the corresponding frames.

To evaluate their approach, the authors introduce a benchmark consisting of 120 images. For each image, a vision-language model (VLM) imagines two to four plausible social interactions. Common camera trajectories are then applied, and the timing of each interaction is randomly sampled. The authors also propose a set of metrics to assess the quality of interaction triggering and generation.

They compare their approach against several baselines using the proposed benchmark and evaluation metrics. In addition, they investigate the effects of training data, the cross-attention mask, and computational cost.

Critical strengths
  • The paper is well-written and easy to follow.

  • The quantitative and qualitative results are convincing and support the effectiveness of the proposed approach.

  • The ability to control when events are triggered is interesting.

Critical weaknesses
  • The paper appears to really close to Warp-as-History (Wang and He, 2026). The claimed contribution of self-distillation seems overstated, and the key technical contribution appears to be the attention mask.

  • The original Warp-as-History paper demonstrates that a video model can be adapted in a zero-shot or one-shot setting. In contrast, the only reported results in this paper use 156 synthetic videos, raising questions about the necessity of this additional training and the model's ability to generalize with limited data.

Questions for authors

My main concern is the novelty of the proposed approach compared with Warp-as-History (Wang and He, 2026).

  • Could the authors (re-)clarify the key differences and novel contributions of their approach relative to Warp-as-History?

  • Could the authors discuss whether their approach, can support zero-shot and one-shot adaptation, as demonstrated in the original work?

Reviewer_TKTvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper, called HelloWorld, introduces a method for making characters say hello or greet the viewer in a video. The method consists of fine-tuning a DiT video generation model based on LTX-2.3, where walkthrough controls can be provided as input, similar to real-time video world models. An additional button, "F", can be used to trigger an action directed toward the viewer. The scene and characters are guided by a first-frame image and a reconstructed scene point cloud. The paper focuses on enabling basic character interaction and timing control by separating camera motion from character interactions through warped-video conditioning. The approach consists of self-distillation using generated example videos and lightweight LoRA training. Temporal cross-attention masking is used to ensure that pressing the F button triggers the interaction within the correct time interval. While the controls are familiar to existing world models, the proposed system is not real-time. Although the control interface is interactive, the image-to-video generation approach requires a reference image and pre-specified camera trajectory and interaction script.

Critical strengths

The paper addresses an important limitation of existing video world models and video generation methods, namely the lack of precise control over character actions, interaction timing, and camera motion during video generation. The paper proposes an approach to improve the controllability of video DiTs through fine-tuning and attention masking. The results look fairly, and the quantitative evaluations also demonstrate solid performance.

Critical weaknesses

My main concern with this paper are the following: socially interactive character are really overclaimed, it just says hello, video world models is also overclaimed it only uses some camera control inputs and has some characters waiving at the viewer. The system is not real-time, and even if it was it requires scene-specific camera trajectory and interaction script, which significantly limits its application and generalizability.

Questions for authors

No additional questions.

Reviewer_eDTSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HelloWorld for generating viewer-directed character interactions with camera and timing control. It finetunes LTX-2.3 on self-generated interaction videos using warp-video conditioning, then applies temporal cross-attention masks during inference. The authors introduce HelloWorldBench and report baseline comparisons, component ablations, and a user study.

Critical strengths
  • Meaningful combination of controls: Jointly controlling camera motion and viewer-directed character actions is a useful extension of camera-controlled video generation. HelloWorldBench evaluates both capabilities across human and non-human subjects.
  • Effective adaptation with limited data: Using only 156 synthesized videos, the method improves ActAcc from 36.4 to 41.4 and reduces GazeDev from 51.3° to 40.2° relative to real-video training, while preserving camera control and visual quality. This supports the value of interaction-rich training data.
  • Supporting ablations and human evaluation: Temporal masking improves TimeAcc from 36.7% to 81.7%. The user study also favors HelloWorld in action naturalness, perceived interaction, and scene quality, providing evidence beyond automatic metrics.
Critical weaknesses
  • Limited methodological advance over closely related work: Camera conditioning follows Warp-as-History [1], while time-windowed interaction masking closely overlaps with SocialDirector [2]. The interaction-rich finetuning is useful, but the methodological contribution largely consists of adapting and combining existing mechanisms. The methodological differences from prior works deserve explicit discussion.
  • Limited evidence for precise temporal control: TimeAcc identifies only whether interaction occurs early, midway, or late. It does not measure action onset/offset errors or unwanted actions outside the requested window. The results therefore establish coarse scheduling, but not the claimed precise localization.
  • Insufficient evaluation of character-specific control: HelloWorld uses prompts to specify who acts, but discusses and evaluates this control only minimally. Unlike SocialDirector's spatiotemporal masking, its mask controls only timing. The paper does not explain this design choice or establish whether prompt-based actor selection is reliable when multiple characters are present.
  • Limited real-time interactivity: The method requires pre-specified interaction windows and camera trajectories, and takes approximately 60 seconds to generate a 10-second video. It therefore demonstrates offline controllable generation rather than real-time interaction with an ongoing world simulation. This is already stated in the limitation part by authors.
Questions for authors
  • What distinguishes the temporal masking from SocialDirector, and what technical contribution goes beyond combining it with Warp-as-History-style conditioning?

  • Are measurements available for action onset/offset accuracy and unwanted interactions outside the specified window?

  • Why use temporal-only masking rather than spatial and temporal constraints as in SocialDirector? What evidence shows that prompts reliably select the intended character, and does ActAcc penalize actions performed by the wrong character?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.