acceptodds
Under review as a conference paper at ICLR 2027

Navigating User Behavior toward Personalized Multimodal Generation

Abstract

Modern AIGC pipelines deliver high-fidelity images and videos but presuppose a well-formed creation instruction, while end users rarely articulate visual details, so personalized content generation must turn a user's interaction history into an executable instruction that aligns downstream synthesis with user demand. Serializing behavior is largely settled—discrete item identifiers from generative recommendation already provide a serviceable interface—so the open obstacles lie in learning: the desired instruction is unobserved and underdetermined, and no signal ties a written instruction back to user intent. We therefore propose NaviGen. An evolution-guided induction phase manufactures its own supervision, using evolutionary search to discover history-to-instruction traces from which the model distills preference reasoning and instruction writing. A closed-loop alignment phase then optimizes a triangular self-consistency reward that couples the written instruction, the model's own preference prediction, and the target semantics, so instruction quality earns credit only when it is user-grounded. Experiments across diverse domains show that NaviGen improves personalized image and video generation and yields more specific, relevant, and visually generatable instructions, with next-item prediction reported as an intermediate diagnostic. Our code is released at: https://anonymous.4open.science/r/NaviGen-08F8/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Personalized Content Generation, Preference Modeling, Supervised Fine-Tuning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Ga9TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

NaviGen turns a user's interaction history into a textual AIGC instruction, which a frozen text-to-image/video model renders. Training has three stages: cold-start initialization, full finetuning, and GRPO. Inference is a single policy pass. Evaluation covers three domains on image and video with four automatic metrics, ablations, human evaluation, and a cost study.

Critical strengths

As a researcher in AIGC, I don't think this paper has any strengths in visual generation. I am not recommendation system experts, so I would like to see the reviewers' comments regarding the recommendation system for further evaluation.

Critical weaknesses
  1. Generation is never shown to be better than retrieval. Nothing compares generating a product image against simply retrieving the product's real image — and generation is slower and typically lower-quality than real images, so the burden of proof is on the paper and is not met.

  2. The interface to the visual generator is text-only. It should be multimodal (text prompt + visual reference image/video).

  3. Relevance leaks the target. The anchor is the mean of history and target caption embeddings, so "the instruction matches the next item" and "the instruction matches this user's taste" score identically — and the model is explicitly trained to predict the target TID. There's no target-free variant to separate the two.

  4. "Evolutionary search" oversells what it is. R=1, population of 3: three founders, one crossover/mutation round, keep the best. There's no comparison to cheaper ways of manufacturing the same supervision (best-of-N, self-refine, DPO over candidate pairs), so I can't tell whether evolution specifically is doing anything.

  5. The two stages push in opposite directions on the same thing. The search prompt penalizes any candidate that "explicitly mentions, copies, or quotes the target item text or tokens", while the RL reward contains R_ins↔t̃ requiring the instruction to anchor to the target TID. Depending on how "anchor" is computed, this either contradicts SFT outright or is badly underspecified.

  6. Split is by instance, not by user or time. An 8:1:1 split of (history, target) pairs is described; if a user contributes multiple pairs, later pairs can land in train while earlier ones land in test. The paper doesn't say how many instances each user contributes.

Questions for authors

Q1. Please explain the advantage of attaching a visual generator over the existing retrieval-based approach.

Q2. Please separately state what your contribution and your difference is, with respect to (i) recommender systems, (ii) preference learning, and (iii) multimodal visual generation.

3 comments

Dear Reviewer Ga9T,

Thank you very much for your time and for the thoughtful feedback. We have carefully addressed all comments in our response and revised our explanations accordingly.

Should there be any further questions or concerns, we would be very happy to provide additional clarification or address them during the discussion period.

Thank you again for your consideration.

Best regards,

The Authors

I thank the authors for the detailed reply.

W6 is resolved. One instance per user with disjoint user sets rules out the leakage I raised, and I withdraw that point.

The three central concerns remain.

  • W1 (generation vs. retrieval). The reply offers a qualitative argument for why retrieval may have nothing to return, but runs no retrieval baseline and reports no count of how often a target lacks a usable asset. The owner scores offered as validation are absolute ratings with no retrieval comparison, and the video means (2.487 / 2.386 / 2.154) all fall below the scale midpoint, which the reply does not discuss. The burden of proof I raised is therefore still unmet.

  • W2 (text-only interface). The reply states that a reference frame is "often unavailable", while W1 states that a catalog packshot exists. It defends the 0.51s policy pass — a figure that excludes the generator dominating end-to-end latency — and concedes that a reference image is "a natural extension". This reads as accepting the limitation rather than answering it.

  • W3 (Relevance). My concern was that the metric mixes the target caption into its anchor, not that the target is an input at test time; no target-free Relevance variant is given. The reported Jaccard of 0.023 (59% of cases sharing no term) shows history and target are semantically distant, which supports rather than refutes the reading that the model predicts a next item instead of translating a preference.

On Q2. The reply confirms that the contribution is not in visual generation (the image/video model is not trained, "the contribution stops at the instruction") and not in recommender architecture (next-CID is "only a diagnostic"). If so, the work has to be judged on whether producing an instruction beats retrieving an asset, and that comparison is still missing. The rebuttal also introduces "plain-text CoRRe" without definition, numbers, or a table; W5 attributes triangular alignment to the judge whereas Appendix A.3/A.5 restricts the judge to the instruction-side quality term; and describing the Relevance drop under w/o GRPO as "holding the instruction on the user" conflates target alignment with user alignment — the same conflation W3 concerns. Since the authors themselves frame the recommendation side as a diagnostic, I no longer consider a recommender-system expert's opinion necessary for evaluating the core claim.

I therefore maintain my rating of weak rejection, and my Soundness score.

Thank you for the follow-up and for confirming that the split concern is resolved. We would like to clarify the remaining points more directly.

Generation and retrieval. The practical distinction is between finding an existing asset and creating a new visual concept. A product packshot may exist without depicting the scene or composition a user would enjoy. NaviGen learns to express such interests as executable instructions. Our experiments test this capability against other personalization methods with the generator fixed; they do not establish superiority over retrieval. For example, Games image Novelty and Aesthetic improve from PROSE's 0.5796/0.7804 to 0.6479/0.8423, with complementary support from the 46-participant comparison. We agree that the owner video means of 2.487/2.386/2.154 are below the scale midpoint. These indicate that the current video experience needs improvement. We will discuss them explicitly and distinguish this preliminary evidence from the stronger image results.

Text-only interface. Our setting studies personalization from behavioral and textual evidence without requiring visual references. Reference images remain valuable for preserving product identity and appearance, and our experiments do not establish that they are unnecessary. We will state this input assumption clearly. The 0.51-second figure measures instruction-generation overhead only, excluding rendering, and does not justify ruling out visual conditioning.

Relevance. You are right that the concern is about the metric's anchor. Mixing history and target embeddings does not separate historical preference from next-item agreement. We will describe the metric as target-conditioned Relevance and interpret the GRPO ablation accordingly. The lexical Jaccard statistic does not resolve this issue. Target-blind Engagement and owner ratings provide complementary human evidence, but a separate target-free automatic evaluation has not been reported.

Reward computation and contribution. We confirm that the LLM judge scores both instruction quality and all three triangular alignment terms. We will correct the inconsistent appendix wording. Alignment measures semantic agreement, whereas the search penalty discourages copying the target label; both aim to preserve the intended concept. The reward is a weighted sum, however, so it encourages agreement rather than enforcing a hard condition, and the shared judge leaves room for correlated biases.

The methodological contribution is learning the behavior-to-instruction mapping through synthesized supervision, distilled reasoning, and joint reward optimization. Keeping the renderer fixed lets us evaluate this learned control without changing the visual backbone. We will also make the evidence boundaries clear: the evolutionary-round study measures offline teacher scores, not superiority over compute-matched best-of-N or self-refinement, and the reasoning ablation does not isolate evolutionary operators. The preliminary CoRRe comparison uses a different split assignment and will not be used to substantiate the main claim.

Thank you for helping us sharpen both the contribution and the interpretation of the results.

Reviewer_bZKVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper focuses on personalized generation and aims to transform users’ interaction histories into high-quality instructions for downstream content generation. Through a carefully designed multi-stage training paradigm, it provides a solution to the challenge of directly learning high-quality generation instructions and ultimately achieves strong performance across multiple benchmarks.

Critical strengths
  1. The application scenario addressed in this paper is meaningful. Translating users’ interaction histories into high-quality generation instructions could benefit a wide range of everyday users of AI tools.

  2. The proposed multi-stage training paradigm is natural and logically coherent. The paper provides generally sufficient ablation studies and relatively detailed experimental and implementation information, which strengthens the credibility and the reproducibility of the results.

Critical weaknesses

None.

Questions for authors
  1. In the full-parameter finetuning stage, the trajectory construction for the CID2INS task appears fairly complex. It would be helpful if the authors could provide a clearer description of how a complete trajectory is organized, including its different components and their roles in training.

  2. The paper uses only one evolutionary round (R=1R=1). An ablation study over different values of RR would help more clearly establish the contribution of the evolutionary instruction-generation module.

  3. In the GRPO stage, RqualR_{\mathrm{qual}} evaluates four dimensions and RalignR_{\mathrm{align}} contains three terms. A component-wise leave-one-out ablation of these reward terms would help clarify their individual contributions and assess the risk of reward hacking. Besides, the paper does not explain how the three components of RalignR_{\mathrm{align}} are computed. It would be better if the authors could provide corresponding definitions.

  4. In the GRPO stage, the paper does not report the mixing ratios of the two tasks and the multi-task training strategy (such as sampling according to a fixed ratio). It would strengthen the paper if the authors could provide further clarification and an ablation over different task proportions.

  5. After the initialization stage, the CID tokens are expected to have acquired a relatively stable representation. It would therefore be informative to report the performance obtained when the CID-related parameters are frozen during subsequent training stages. Together with the existing ablations, this comparison would help more precisely characterize the contribution of the initialization stage and whether later training causes representation drift.

  6. I wonder whether NaviGen’s specialized training comes at the cost of degraded general capabilities. It would help address this concern if the authors could report the model’s performance on general QA or instruction-following benchmarks.

Since the rebuttal period is limited, if the authors are unable to address all of the questions above, I would encourage the authors to prioritize Questions 1, 2, and 3.

Reviewer_JhUaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper explores personalized content generation: given a user’s interaction history, the model generates a tailored prompt for text-to-image or text-to-video generation. Rather than directly using text representations, NaviGen represents each item using a three-level residual-quantized code (CID) and an LLM-compressed set of keywords (TID). During training, the model learns to map between TIDs and CIDs while generating target instructions derived from an LLM-judged search over candidate prompts, followed by GRPO fine-tuning. At inference, NaviGen predicts a final instruction in a single pass over the user's CID history. Experiments and ablation studies on two Amazon domains and a short-video benchmark demonstrate the effectiveness of the proposed approach.

Critical strengths
  • Efficiency Advantage: Inference is highly efficient, requiring only a single two-stage pass (~0.51s with a 1.7B model).

  • Training and evaluation judges are sourced from distinct model families to mitigate bias. The authors explicitly acknowledge the lack of a judge-calibration study and provide full code and prompt templates for reproducibility.

Critical weaknesses

Task and Deployment Scenario

NaviGen relies exclusively on user interaction history without incorporating task-specific constraints or user intent. Consequently, the framework is limited to passive content recommendation and cannot generate target-driven outputs. On the merchant side, E-commerce sellers cannot combine a specific target product with user preferences to render tailored marketing assets. On the user side, end-users have no control mechanism to guide or modify the generated content. This unconstrained design contradicts the paper's stated motivation of supporting downstream applications like targeted advertising, e-commerce, and poster generation.

Evaluation Claims

Evaluating open-ended creative outputs against a single ground-truth target item is fundamentally misaligned with the task. Future user interest is intrinsically open-ended (e.g., in Figure 6, romantic, master–apprentice, or fantasy directions are all valid). While Section C1 notes that optimal instructions are underdetermined, the paper evaluates against a single next item.

Unjustified Interface Design and Poor Generalization

  • Missing Plain-Text Baselines: Introducing new CID tokens requires an extra pre-training stage, yet there is no direct comparison against passing history as plain text (e.g., using TIDs directly). Furthermore, the input representation for the "w/o CID" ablation is left unspecified.
  • Unclear Out-of-Distribution (OOD) Handling: The paper does not evaluate generalization to unseen items, cross-platform data, or new languages. Although the Taobao merchant study represents a domain shift, the method for processing these external histories is omitted.

Vulnerability to Noisy and Heterogeneous Histories

Real-world interaction logs contain noisy, non-creative interactions (e.g., daily groceries or household items). Unlike reference-guided frameworks (e.g., Pigeon, RAGAR) that dynamically filter or weight interactions relative to a target context, NaviGen ingests the raw, unfiltered history. Because all evaluated datasets are strictly domain-homogeneous, the system's robustness to noisy real-world data remains untested.

Questions for authors
  • In the owner study, how were Taobao items mapped to CIDs and TIDs, given that the codebook never saw them and their captions may be in Chinese? Which domain’s model was used, and how were the self-rating means computed?
  • Several reported numbers appear inconsistent. For example, CIPHER and TRIPLE have identical image Relevance on all three datasets in Table 2 (0.4273 / 0.4038 / 0.3963), and the Short Videos R@20 column in Table 4 is identical to the Games column. Could you check the tables?
4 comments

Dear Reviewer JhUa,

Thank you very much for your time and for the thoughtful feedback. We have carefully addressed all comments in our response and revised our explanations accordingly.

Should there be any further questions or concerns, we would be very happy to provide additional clarification or address them during the discussion period.

Thank you again for your consideration.

Best regards,

The Authors

Reviewer_JhUaOpenReview ↗

Response to authors' rebuttal

Thank you for the detailed rebuttal. Some of my concerns or questions are resolved, such as how the Taobao data are processed. However, two of my core concerns are not directly addressed yet. New experiments are not required, but I would like to see more convincing arguments.

  1. Motivation and application. In the rebuttal, the authors clarify the scope of the task. But the value itself is not discussed yet. What concrete value does it bring to users? Take browsing Taobao or Amazon as an example. The features mentioned in the rebuttal, such as some tabs we would see like “guess you like”, all present concrete products that users can actually buy. How does the proposed method, which generates content for a predicted item, compare with these features?

  2. Evaluation claim. The rebuttal does not respond to my core concern itself. In the rebuttal, the authors say that the system has to commit to one next interest, but this does not mean that there is only one correct answer (the next interest can be various and they can all be plausible); the rebuttal mentions that other directions can also be coherent. In addition, the previous works cited in the rebuttal evaluate with top-K lists, where the prediction counts as correct if the ground-truth item is among the top K. Please correct me if I am wrong:

  • SASRec (Kang and McAuley, ICDM 2018), p. 7: “Hit@10 counts the fraction of times that the ground-truth next item is among the top 10 items.”
  • TIGER (Rajput et al., NeurIPS 2023), p. 5: “We use top-k Recall (Recall@K) and Normalized Discounted Cumulative Gain (NDCG@K) with K = 5, 10 to evaluate the recommendation performance.”

Could the authors please explain a little bit more on this?

Thank you for clarifying these two points.

Practical value. NaviGen aims to make personalized visual creation easier for users who have interests but cannot readily express them as detailed prompts. For example, a history of interest in fantasy games could inform an illustration with a suitable setting, composition, and atmosphere. The benefit is turning implicit preferences into usable creative instructions. Product recommendation serves the related purpose of finding existing items to buy; our current setting concerns visual creation and inspiration. We will clarify this distinction and avoid suggesting that history-only generation already supports faithful advertising for a specified product.

Evaluation. You are right that SASRec and TIGER evaluate top-K lists. Our earlier response confused one observed positive item with one valid output. Several creative directions can suit the same user. The held-out next item provides one observed preference signal, so Relevance measures compatibility with observed behavior rather than all acceptable creations. It is a continuous score using both history and target semantics, separate from the top-K CID prediction diagnostic. A plausible alternative can score lower, which is a limitation we will state explicitly.

We therefore also include target-blind Engagement, where raters see the history and generated content but not the target. In the 46-participant comparison, NaviGen receives the highest mean ratings on all three dimensions for both modalities. Together with the owner study, this offers complementary evidence beyond target agreement, while broader deployment value remains to be tested.

Thank you for helping us make the motivation and evaluation clearer.

Thank you to the authors for the follow-up!

Regarding the first point, the current scope definition makes sense to me now. Please update them in the revised manuscript.

Regarding the second point, I still anticipate a more reasonable benchmark score, and I believe the authors can develop a better evaluation metric. However, given that the rebuttal deadline is approaching and the authors have clearly put a lot of effort into this work, I am raising my score to a weak accept (leaning toward borderline, 2.5).

Good luck!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.