acceptodds
Withdrawn from ICLR 2027

WorldWeaver: Benchmarking Multimodal Intent Recognition and User Modeling in Interactive Visual Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Personalized interactive continuous image generation is an important task for adaptive visual information distribution, in which visual content evolves with user intent and preferences. Existing benchmarks typically study final-image personalization, multimodal intent recognition, and user modeling separately; they rarely represent generated visuals as environments for later feedback or preserve alternative continuations from a shared history. We introduce WorldWeaver-Bench, an extensible benchmark for this setting. Its shared-prefix branching trees support joint evaluation of multimodal interaction-intent recognition, persistent user-profile modeling, and matched comparisons of alternative paths. Experiments show that story context improves the interpretation of current interaction intent, longer and broader interaction evidence improves profile recovery, and preference-compatible paths are not consistently the most diagnostic for user modeling. WorldWeaver-Bench provides a testbed for studying how generated visual content both responds to user preferences and reveals them through interaction.

Primary area:
datasets and benchmarks
Keywords:
User Modeling, Interactive Image Generation, Multimodal Intent Recognition, Story Context, Shared-Prefix Trees

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_WCvVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces WorldWeaver-Bench, a synthetic benchmark for multimodal interaction-intent recognition and user-profile modeling in interactive visual narratives. It contains 82 simulated users, 12 topics, 328 user-topic trees, and 6,778 intent examples. A model-assisted construction pipeline generates profile-conditioned interactions and successor images, preserving alternative continuations from shared prefixes. The two primary tasks predict a six-class interaction intent and recover an initial eight-dimensional user profile from observed histories. Experiments compare supervised and prompted models, examine story context and evidence coverage, and retrospectively compare profile recovery along higher- and lower-preference-propensity paths.

Critical strengths
  1. Shared-prefix organization permits controlled starting conditions. Retaining alternative continuations for the same user and history is a useful design for comparing the evidence available along different paths. This provides a potential foundation for studying interactive user modeling beyond a single fixed trajectory.
Critical weaknesses
  1. The benchmark lacks independent validation of its simulated users and behavioral targets. The dataset comprises 82 simulated users across 12 narrative topics. Section 3 describes how predefined profiles condition generated interactions, stories, and intent annotations, with the construction model also involved in consistency checking. Profile prediction is then evaluated against the initial synthetic vectors. This establishes recoverability within the construction framework, but does not establish that the profile-to-behavior mapping captures meaningful variation in user behavior. Multiple stories and branches do not by themselves demonstrate behavioral diversity or realistic ambiguity. Section 3.3.4 reports manual inspection without its sample size, annotator agreement, or quantitative outcomes, and there is no validation against real-user behavior or an independent behavioral mechanism. I recognize that the paper explicitly disclaims psychological measurement and real-user claims; the concern is instead whether its operational targets constitute a useful benchmark for the intended user-understanding problem. Related work does provide relevant validation precedents: PIPBench includes human-selected visual preferences, and Yu and Riedl's personalized interactive-story work evaluates real users' story ratings. Synthetic data are not inherently disqualifying, but the current submission does not supply an adequate alternative basis for validating its core resource.

  2. The experiments do not establish sufficient diagnostic value or new knowledge beyond the synthetic setup. The main findings are that supervision improves prediction, preceding story context improves intent classification, and more observed interactions improve profile recovery. These results do not separate visual grounding, narrative interpretation, and persistent preference inference from learning construction-specific textual or behavioral regularities. The story-context ablation in Table 3 establishes the utility of additional narrative information, not specifically user modeling; the final condition in Figure 4 increases both topic coverage and the total interaction budget. The distinctive shared-prefix analysis is also limited: in Table A5, only one of six model/input settings has a 95% interval excluding zero, while the operational definition of preference propensity, path-selection procedure, and treatment of repeated users in interval estimation remain unclear. The other intervals are inconclusive. Consequently, the paper does not convincingly demonstrate which model capabilities or failure boundaries its additional structure reveals. Given that multimodal intent recognition, profile-conditioned generation, and preference-based branching narratives already have substantial precedent, combining these elements requires stronger evidence of a useful new diagnostic contribution than is currently provided.

Questions for authors
  1. What independent evidence validates the relationship between the initialized profile dimensions and the generated behaviors? Is there any existing comparison with real-user choices, independent human judgments of profile-behavior compatibility, or a separately constructed behavior model? Please distinguish such validation from checking internal consistency against the generating model's own plan.

  2. For the manual audit in Section 3.3.4, how many records and users were examined, by how many annotators, and with what agreement and acceptance rates? In particular, how reliably can independent annotators distinguish overlapping intent categories such as reveal and interact?

  3. What evidence shows that profile recovery uses visual interaction evidence and persistent user tendencies beyond story-text regularities? Are matched story-only and multimodal results available, and do they support a capability-specific interpretation of the reported gains?

  4. How is preference propensity defined and computed, how are the 64 path pairs selected, and how are the Table A5 confidence intervals obtained? Do the intervals account for repeated observations from the same 16 users, and does the positive result remain reliable under that treatment?

Reviewer_CLy5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WorldWeaver-Bench, a benchmark for multimodal intent recognition and user modeling in personalized interactive visual generation. The benchmark consists of simulated users, narrative topics, and shared-prefix branching interaction trees. It provides structured annotations for interaction intents, user profiles, and visual narrative states. Experiments evaluate multimodal intent recognition, long-term user profile modeling, and the relationship between user preferences and the diagnostic value of alternative interaction paths.

Critical strengths
  1. This paper investigates personalized interactive visual generation, which is an interesting and practically meaningful research direction.

  2. The proposed benchmark introduces shared-prefix branching interaction trees, enabling controlled comparisons of alternative interaction paths. It also provides annotations for interaction intents and user profiles.

  3. Several experiments are conducted to evaluate multimodal intent recognition and user profile modeling, including comparisons with different model architectures and ablation studies.

Critical weaknesses
  1. The benchmark relies entirely on simulated users and synthetic interactions, which raises concerns about its generalization to real-world scenarios. The user profiles are predefined, and the interaction trajectories are generated based on these profiles. Therefore, models may learn the behavior patterns of the simulation process rather than generalizable human preferences. Some validation using real users or independently generated interactions would strengthen the practical value of the benchmark.

  2. The reliability of the dataset annotations is not sufficiently evaluated. Although Section 3.3.4 describes automated quality checks and manual inspection, the paper does not report the number of manually inspected samples, annotation accuracy, or inter-annotator agreement. Since the benchmark relies heavily on automatically generated interaction intents and user states, more quantitative analysis of annotation quality should be provided.

  3. Although the benchmark focuses on multimodal interaction understanding, the experiments lack comprehensive comparisons between text-only, image-only, and multimodal inputs. Table 3 only examines the effect of story context. More ablation studies are needed to demonstrate the importance of visual information for intent recognition and user modeling.

  4. As shown in Table 4, different methods use different additional information. For example, PINet-adapted uses story context and intent information, while MPRNet-adapted uses neither, and Qwen-based methods use story context. Comparisons under the same input conditions would be more convincing.

  5. Although the dataset contains 6778 intent examples, it includes only 82 simulated users and 12 narrative topics, with 16 users in the test split. More experiments on unseen topics and diverse user profiles would help demonstrate the generalization ability of the benchmark.

Questions for authors

Please see Weaknesses.

Reviewer_Ui57OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces WorldWeaver-Bench, a novel benchmark designed for personalized interactive continuous image generation. Unlike traditional single-image personalization or static intent recognition benchmarks, WorldWeaver-Bench frames generated visual content as an evolving environment for ongoing user feedback and interaction. It organizes user-topic interactions into shared-prefix branching trees. The benchmark supports evaluation across two primary tasks: Classifying user visual/narrative modification intents into 6 taxonomy categories (zoom in, reveal, branch out, reframe, follow, interact). And recovering a user's initial 8-dimensional long-term preference vector from observed interaction histories. The dataset is constructed via an automated pipeline using 82 simulated users, 12 narrative topics, 328 user-topic trees, and 6,778 intent examples.

Critical strengths
  1. Representing continuous generation as shared-prefix branching trees allows for matched comparisons of alternative paths originating from identical interaction histories.

  2. The benchmark integrates multimodal intent grounding with a structured user-state representation based on Yee's game motivation framework and PAD/SAM emotional models.

  3. The experiments demonstrate that preceding story context improves localized intent disambiguation, and preference-compatible paths are not necessarily the most diagnostic for user modeling.

Critical weaknesses
  1. Lack of Strong Closed-Source SOTA Models (e.g., GPT-6.x, Claude 5.x). The evaluated baselines are currently limited to open-source or open-weight models (e.g., Qwen3.5-9B/27B, Qwen3-VL, InternVL3.5-30B, Ministral-3-8B) alongside lighter visual encoders (CLIP, SAM-CLIP, ViLT, CAGC, MVCL-DAF, PINet, MPRNet). The absence of top-tier commercial MLLMs leaves a significant gap in understanding the true performance upper bound of state-of-the-art vision-language models on this benchmark.

  2. Fully Synthetic Dataset and Sim-to-Real Gap. All 82 users, user profiles, short-term states, narrative choices, and interaction histories are simulated using an LLM (gpt-5.5). All visual scenes are synthesized using an image generation model (gpt-image-2). Because the benchmark is 100% synthetic without human participants or human-in-the-loop behavioral validation, it risks evaluating "AI predicting AI-simulated behavior". The paper lacks analysis on how well these simulated behavior policies reflect real human interaction patterns and noisy preference dynamics.

  3. Repository Lacks Dataset Samples and Artifacts. Without downloadable dataset samples or visual inspection files during the review process, reviewers cannot independently verify the visual consistency, prompt fidelity, or annotation quality of the generated branches.

  4. In continuous visual generation, image drift and character inconsistency are critical failure modes. The benchmark focuses on intent accuracy and profile error but omits quantitative metrics (e.g., CLIP/DINO feature similarity across sequential nodes) to evaluate the visual stability of the generated environment.

Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.