acceptodds
Under review as a conference paper at ICLR 2027

V-Engram: Trigger-Indexed External Memory for Modular Text-to-Image Personalization

Abstract

Pretrained text-to-image models contain broad visual knowledge, yet they cannot reliably acquire or refine a specific visual identity from only a few references while preserving compositional control. Token-embedding methods are compact but often underfit identity, whereas adapter-based methods improve fidelity through persistent weight updates that can be costly to store and interfere when concepts are composed. We introduce V-Engram, a trigger-indexed external memory mechanism for Stable Diffusion 3.5. Each concept is assigned an explicit trigger that retrieves concept-specific memory, whose gated directions enter frozen text-encoder and MMDiT context states as relative residuals. Separating this memory from backbone adaptation enables prompt-selective and multi-concept access without merging model updates. Experiments show that V-Engram broadly matches DreamBooth-LoRA in overall subject fidelity while showing advantages in settings such as contextual subject preservation. Prompt-matched loading retrieves only matched entries, reducing most additional adaptation-state loading for a single-concept query. Qualitative results further demonstrate paired-trigger composition and same-class separation, while prompts without registered entries retain the frozen model's base behavior. Together, these results establish trigger-indexed memory as a modular interface for adding targeted visual evidence without rewriting the generator.

Primary area:
generative models
Keywords:
Text-to-Image Personalization, External Memory, Diffusion Models, Few-Shot Learning, Parameter-Efficient Adaptation, Compositional Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TNUUOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes V-Engram, an optimization method for subject-based customization for text-to-image diffusion models. The proposed method aims to introduce trigger words for the specialized concepts, which try to optimize a set of associated modulation parameters, which brings the ability of modifying both the represnetations in the text encoders of the diffusion model along with the MMDiT. The authors demonstrate the effectiveness of their method on Stable Diffusion 3.5, and compare with optimization based methods Textual Inversion and DreamBooth-LoRA. Overall, the authors show the effectiveness of their method uboth qualitatively and quantitatively, where they match the performance of DreamBooth-LoRA without intervening the model weights and surpass Textual Inversion. In addition, the authors also show the abilities if their method in terms of modelling detailed concepts such as human faces and generating multiple concepts in a given generation.

Critical strengths
  • The proposed method offers a new optimization method for per-concept customization of text-to-image generation models, which proposes learned modulation artifacts. In presented qualitative and quantitative comparisons the method apper to be effective in reflecting the concepts in generation.
  • Shwoing the flexibility of the method, the authors demonstrate the effectiveness of their approach on multiple concepts, and human identities, which are subjects with high level of detail.
  • The paper provides the experiment setup in detail, which makes reproducibility possible.
Critical weaknesses
  • The paper is mainly designed as a Stable Diffusion 3.5 oriented approach. While the concept of defining engrams for a trigger token can be generalized into multiple generators, the paper has no such demontsrations and the results are shown in a single model. Furthermore, the methodology appears to be a model specific approach, which reduces the claim of novelty in the proposed method.
  • Overall, the definitions provided in Sec 3.2 are not coherent and makes understanding the method difficult. It appears that the method attempts to learn injectable activations per text encoder with the flow matching objective, however this is not straightforward to the reader in the papers current form where I, as a reviewer, was able to understand this with the help of Figure 2 and with the familiarity on how DreamBooth-LoRA is trained. Furhtermore, the paper needs a revision in terms of clarity.
  • While the method compares with methods such as textual inversion and DreamBooth-LoRA, the comparisons miss a family of methods that use external adapters and image conditioning for personalization, which also serve as common workflows for customized content creations. Additionally, in the included comparisons, prompt following comparisons only cover DreamBooth-LoRA for comparison, which provides a narrow comparison with closely reported metrics. Additionally, to resolve the tension between the subject alignment and prompt alignment, the authors are advised to run user studies which can provide a better assessment for a joint objective, which is the main goal of personalization.
  • This is not very critical but can be corrected in a revision: In Fig. 1, the captions seem off from the actual generations, which makes it harder to identify the strength of the method on prompt alignment. A presentation that aligns the text captions and the generations would benefit the presentation.
Questions for authors
  • Is the method transferrable to other MMDiT based models (e.g. FLUX-1.dev), if not can the limitations be shown over examples or discussed?
  • Can the authors provide comparisons with adaptor based methods, in addition to optimization based methods? Such comparisons can be better suited with identity specific generations (as presented in Figure 8 and Table 4), where concepts have more details and can demontsrate the V-engram's utilization of activation space better.
  • For evaluations, can the authors provide user studies to resolve the tension between prompt following and subject following?
3 comments
AuthorsOpenReview ↗

Cross-Backbone Validation and Joint Composition Evaluation

Thank you again for your helpful feedback. We have uploaded a revised manuscript addressing the questions you raised.

  1. Transfer to other backbones. Yes. We adapted the memory mechanism to the SD1.5 U-Net and the FLUX.1-dev transformer. With memories trained separately for each backbone, enabling memory improves CLIP-I and DINOv2 on both (Sec. 4.5).
  2. Method clarity and Figure 1. We clarified how memories are trained, retrieved by matching registered token sequences, and injected as residuals (Sec. 3). We also updated Figure 1 and its caption so the descriptions match the displayed generations.
  3. Broader comparisons and the portrait study. We added SD3.5 adaptations of TARA and PureCC, as well as FLUX.1-Kontext-dev as a reference-conditioned baseline (Sec. 4.2). The original portrait results showed encouraging identity similarity in the tested cases. However, that study was intended to probe refinement of existing knowledge and did not isolate that effect. We removed it to focus the evaluation on subject memory, composition, and transfer across backbones.
  4. Joint evaluation. We did not conduct a human study within the rebuttal period. We instead added an automated Qwen3.5-Plus evaluation on matched two-subject prompts. The joint criterion requires both identities and all prompt conditions to pass. On this set, V-Engram achieves 56.7% joint success versus 3.3% for DreamBooth-LoRA (Sec. 4.4, Table 3).

We took your feedback seriously and carried out additional experiments to address the concerns you raised. We believe the new results make the value of V-Engram clearer: it allows subject memories to be stored independently, retrieved on demand, and used together in a single generation. If the revised manuscript addresses your main concerns, we would be grateful if you would consider raising your overall rating.

Reviewer_TNUUOpenReview ↗

Rebuttal Response by Reviewer TNUU

I appreciate the authors' detailed response to the reviewers' concerns and the additional experiments presented in the revised manuscript. From my perspective, the concerns regarding generalizability beyond Stable Diffusion 3.5, methodological clarity, limited baseline comparisons, and the caption inconsistencies have been substantially addressed. Considering these improvements, I am raising my score from weak rejection to weak acceptance.

However, I would like to highlight that the method still exhibits a trade-off between subject fidelity and prompt alignment. (reflected in the lower contextual CLIP-T score compared to other methods, see Table 1). Additionally, while I acknowledge that conducting a human evaluation within the limited rebuttal timeframe is challenging, I believe such a study would be a valuable addition to the final version, in case of acceptance, to better assess the balance between subject preservation and prompt following.

AuthorsOpenReview ↗

Thank You for Your Reassessment and Helpful Suggestions

Thank you for your thoughtful reassessment and for recognizing the improvements in the revised manuscript. We sincerely appreciate your constructive guidance throughout the discussion.

We agree that the trade-off between subject fidelity and prompt alignment remains important. In the latest revision, we have added Qwen assessments of identity preservation, prompt compliance, and joint success across the main baselines in Table 1, and revised the discussion to characterize this trade-off more explicitly.

We also appreciate your suggestion regarding human evaluation. Such a study would provide valuable complementary evidence, and we will carefully consider it for the final version, should the paper be accepted.

Thank you again for your time and helpful feedback.

Reviewer_6FF4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to learn specific subjects from a few images while keeping them recognizable in different contexts. V-Engram uses prompt triggers to retrieve learned memories and inject them into a frozen SD3.5 model. Experiments show similar subject fidelity to LoRA, with better subject preservation but weaker text alignment in new contexts.

Critical strengths
  1. The paper addresses a useful task of learning specific subjects from a few reference images while preserving their identity across different contexts. The motivation is clear and highlights the need to balance subject fidelity with compositional control.

  2. The method has a clear modular structure. Concept memories are stored separately from the frozen backbone and activated through prompt matching. This allows selective memory loading and preserves the original model’s behavior when no registered entries match.

Critical weaknesses

I have significant concerns about the experimental evaluation.

1.The comparisons for both single-concept and multi-concept customization are too limited, making it difficult to assess the advantages of the proposed method over existing approaches. Several relevant methods have publicly available code, including CIDM [1], LoRA-S [2], Mix-of-Show [3], PureCC [4], and B-LoRA [5]. The authors should include comparisons with these methods in the relevant settings to better support their claimed contributions.

  • [1] Jiahua Dong et al. How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization? NeurIPS, 2024.
  • [2] Ming Zhong et al. Multi-LoRA Composition for Image Generation. Transactions on Machine Learning Research, 2024.
  • [3] Yuchao Gu et al. Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models. NeurIPS, 2023.
  • [4] Zhichao Liao et al. PureCC: Pure Learning for Text-to-Image Concept Customization. CVPR, 2026.
  • [5] Yarden Frenkel et al. Implicit Style-Content Separation using B-LoRA. ECCV, 2024.

2.The ID customization experiments also include very few baselines, making it difficult to assess the claimed advantages in identity preservation. Relevant portrait customization methods such as PuLID [6], SPF-Portrait [7], and InstantID [8] are missing from the evaluation. The authors should include broader comparisons while accounting for differences in reference inputs and fine-tuning requirements.

  • [6] Zinan Guo et al. PuLID: Pure and Lightning ID Customization via Contrastive Alignment. NeurIPS, 2024.
  • [7] Xiaole Xian et al. SPF-Portrait: Towards Pure Text-to-Portrait Customization with Semantic Pollution-Free Fine-Tuning. arXiv preprint arXiv:2504.00396, 2025.
  • [8] Qixun Wang et al. InstantID: Zero-shot Identity-Preserving Generation in Seconds. arXiv preprint arXiv:2401.07519, 2024.

3.After reviewing the experiments and references, I find the discussion of recent advances incomplete, which raises concerns about the method’s contribution and the validity of the experimental conclusions. The authors should include the missing references and provide appropriate comparisons and analysis to clarify their contribution.

4.The ablation compares the full method with direct single-vector injection, removing direction normalization, hidden-state-relative scaling, and per-channel gating at the same time. This makes it unclear which component accounts for the improvement and does not establish whether gating is necessary. The authors should remove each component separately to evaluate its individual contribution.

Questions for authors

My specific concerns are detailed above. Please add the suggested baseline comparisons and ablation experiments, and update the related work with recent references.

2 comments
AuthorsOpenReview ↗

Additional Baselines, Component Ablations, and Revised Scope

Thank you again for the detailed suggestions. We have uploaded a revised manuscript with the following changes:

  1. Additional baselines. We added TARA and PureCC to the main comparison and PureCC to the seven-dog disambiguation study. In that study, V-Engram achieves the highest closed-set retrieval accuracy under both indexed and name-based triggers (Secs. 4.2 and 4.4).
  2. Scope of the portrait study. The original study was intended to explore whether V-Engram could refine knowledge already present in the pretrained model. The experiment did not isolate that effect, and it was peripheral to the paper’s main contribution. We removed the standalone study and its associated claims, and added experiments on SD1.5 and FLUX.1-dev to test whether the memory mechanism works across different architectures (Sec. 4.5).
  3. Related work. We expanded the discussion of recent personalization methods and clarified how V-Engram differs from token-aware LoRA, optimization-based customization, and reference-conditioned generation (Sec. 2).
  4. Component ablations. We now remove relative scaling and per-channel gating one at a time, keeping direction normalization fixed, and evaluate both variants under the same contextual protocol. Removing either component reduces subject similarity, with the larger drop coming from removing relative scaling (Sec. 4.3, Table 2(b)).

We took your feedback seriously and carried out additional experiments to address the concerns you raised. We believe the new results make the value of V-Engram clearer: it allows subject memories to be stored independently, retrieved on demand, and used together in a single generation. If the revised manuscript addresses your main concerns, we would be grateful if you would consider raising your overall rating.

AuthorsOpenReview ↗

Brief Update on the Revised Manuscript and Additional Evaluation

Dear Reviewer 6FF4,

Thank you again for your detailed and constructive feedback. Following our earlier response, we would like to briefly highlight the completed revisions relevant to your concerns.

The revised manuscript includes SD3.5 adaptations of TARA and PureCC, alongside the reference-conditioned FLUX.1-Kontext-dev baseline, and expands the related-work discussion. We have also clarified the component ablations and included cross-architecture experiments on SD1.5 and FLUX.1-dev. As discussed previously, we removed the standalone familiar-face study and its associated claims to keep the paper focused on concept personalization.

Most recently, we added a unified Qwen evaluation of identity preservation, prompt compliance, and joint success across all five main-comparison methods, including V-Engram and DreamBooth-LoRA. These results are now included in Table 1, with a revised discussion of the fidelity–prompt-compliance trade-off.

We sincerely appreciate the time you have devoted to our submission. We hope these revisions address a substantial part of your concerns, and would be grateful for any further feedback. If you find that the additional evidence strengthens the paper, we would greatly appreciate your reconsideration of the assessment and rating.

Reviewer_dfRhOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents V-Engram, a trigger-indexed external memory mechanism for text-to-image personalization. It freezes SD3.5 and injects concept-specific gated residuals into the text encoders and selected MMDiT layers. Experiments examine subject fidelity, contextual and multi-subject composition, and face identity preservation.

Critical strengths
  1. The design clearly separates concept parameters from the backbone, enabling selective activation and loading while preserving base-model behavior when no entries match.

  2. The multi-layer gated residual injection has some novelty. Ablations show that MMDiT injection improves subject fidelity over text-encoder-only injection.

  3. The method shows promising subject preservation under contextual prompts and good face identity similarity across several personalization settings.

Critical weaknesses
  1. Baseline comparisons are limited, and the advantages remain unclear. Comparisons mainly cover TI-style learning and standard LoRA, omitting methods such as TARA that address token-level adaptation and multi-concept composition. Relative to LoRA, the method requires more training computation while still trading text alignment for identity preservation.

  2. The practical value relative to current reference-image methods is unclear. Experiments are limited to SD3.5, without comparisons to methods requiring no per-subject training, such as open-weight FLUX.1 Kontext and FLUX.2. The paper should clarify what the additional training and storage provide in identity consistency, semantic control, or multi-subject composition.

  3. The evaluation does not sufficiently support the composition claims. CLIP-I and DINO scores do not fully capture fine-grained identity, attribute binding, or relationship correctness, while multi-subject results are mainly qualitative. A capable VLM could provide evaluation of these individual aspects, with human judgments used to check reliability.

Questions for authors

Please address the concerns raised in the weaknesses above.

4 comments
AuthorsOpenReview ↗

Revised Baseline Comparisons and Composition Evaluation

Thank you again for your feedback. We have uploaded a revised manuscript with additional experiments addressing your three concerns.

  1. Baselines and practical cost. We added SD3.5 adaptations of TARA and PureCC (Sec. 4.2). Compared with DreamBooth-LoRA, V-Engram achieves higher CLIP-I and DINOv2 in held-out contexts and better preserves both subjects on the evaluated two-subject prompts. The memory budget grows with the number of registered concepts: the one-concept registry contains 0.639M parameters, compared with 5.895M for the evaluated rank-4 LoRA configuration. V-Engram takes longer per training step in our controlled comparison, but offers a different inference workflow: a text trigger acts as a lookup key, so the system loads only matching memories and can activate several without merging weight updates.
  2. Reference-conditioned comparison and portability. We added FLUX.1-Kontext-dev. Kontext achieves the highest subject similarity under simple prompts, while V-Engram achieves the highest CLIP-I and DINOv2 in held-out contexts among the evaluated methods. Some reference-conditioned systems acquire their conditioning capability through large-scale training for a particular backbone. Our SD1.5 and FLUX.1-dev experiments show that the same retrieval and injection mechanism can be adapted across backbones, with subject memories trained separately from a few reference images (Sec. 4.5).
  3. Composition evaluation. We agree that CLIP-I and DINOv2 alone do not establish composition correctness. We added Qwen3.5-Plus judgments on 20 matched, archived two-subject prompts, with separate criteria for identity, spatial correctness, and joint success. Both identities pass in 70.0% of assessments for V-Engram versus 5.0% for DreamBooth-LoRA (Sec. 4.4, Table 3).

We took your feedback seriously and carried out additional experiments to address the concerns you raised. We believe the new results make the value of V-Engram clearer: it allows subject memories to be stored independently, retrieved on demand, and used together in a single generation. If the revised manuscript addresses your main concerns, we would be grateful if you would consider raising your overall rating.

Reviewer_dfRhOpenReview ↗

Reply to Authors

Thank you for the response and the additional experiments. The new comparisons are helpful.

However, V-Engram has the lowest contextual CLIP-T score in Table 1, which raises the question of whether its higher subject similarity comes at the expense of following the prompt. Please apply the same VLM evaluation to the existing contextual outputs from PureCC, FLUX.1-Kontext-dev, and V-Engram, assessing identity preservation, prompt compliance, and joint success separately. The current VLM evaluation only compares V-Engram with LoRA on two-subject prompts and does not resolve this concern.

Also, TARA specifically targets multi-subject composition, yet it is only included in the single-subject comparisons. Please include TARA in the two-subject evaluation in Table 3 under the same prompts and scoring protocol. This would provide a more relevant baseline for assessing the claimed composition benefits.

AuthorsOpenReview ↗

Follow-up on Contextual Prompt Compliance and TARA

Thank you for raising these two points. To check what the lower contextual CLIP-T score means in practice, we will try applying the same Qwen3.5-Plus evaluator to the existing outputs from PureCC, FLUX.1-Kontext-dev, and V-Engram. We will keep the pass-rate format from Table 3: one judgment for whether the subject matches the references, another for whether the image satisfies the visible conditions in the prompt, and a joint pass only when both criteria are met. This lets us test whether the lower CLIP-T reflects more missed prompt conditions. We will treat the results as automated judgments, since subtle markings, visually similar subjects, and ambiguous spatial relationships can be difficult for Qwen to assess reliably.

TARA was originally developed for the U-Net architectures of SD1.5 and SDXL. For our main comparison, we adapted its token-focus masking to the text key and value projections of SD3.5’s joint attention, using the same seven MMDiT blocks as V-Engram. As Figure 3 shows, the adapted TARA sometimes fails to preserve recognizable subject details even in single-subject generation. By comparison, the V-Engram examples in Figure 8 retain recognizable subject features after adaptation to both SD1.5 and FLUX.1-dev. These figures illustrate the behavior we observed in the respective experiments. Because the TARA port already struggles to preserve individual subjects, a low score in Table 3 would be hard to interpret as evidence about TARA’s ability to compose them. We therefore did not include this port in Table 3, and we do not draw conclusions about native TARA from its behavior on SD3.5.

AuthorsOpenReview ↗

Additional Evaluation Results and Clarification of the TARA Comparison

Thank you for your helpful follow-up and for recognizing the value of the additional comparisons. We have carefully considered both points and further revised the manuscript.

  1. Contextual evaluation. We have added Qwen-based assessments of identity preservation, prompt compliance, and joint success to Table 1, covering V-Engram, PureCC, FLUX.1-Kontext-dev, DreamBooth-LoRA, and TARA under the same scoring protocol. We also report valid-response counts. V-Engram achieves identity preservation close to Kontext, while its lower prompt compliance supports your observation that subject fidelity involves a trade-off with following contextual instructions. We have revised the discussion to make this trade-off explicit.

  2. TARA comparison. We conducted additional multi-concept experiments using the same SD3.5 adaptation configuration as in our main comparison. Consistent with its weak single-subject results, the adapted version also performed poorly in this separate multi-concept test. We therefore did not include it in Table 3, as the comparison appears strongly affected by the limitations of our adaptation and would provide limited insight into native TARA’s composition capability. TARA was originally developed for U-Net-based generators, whereas our implementation transfers its token-focus masking and alignment objective to MMDiT. We do not interpret these results as evidence against the original method. We acknowledge that this separate test does not replace the exact matched comparison you requested.

This experience also highlights our motivation to explore a memory mechanism applicable across generator architectures. Our final experiment demonstrates its feasibility on SD1.5 and FLUX.1-dev, with memories trained separately for each backbone.

We sincerely appreciate your time and constructive guidance throughout the discussion. We have made a substantial effort to address your concerns and clarify both the contribution and its limitations. We believe the revised work offers a meaningful perspective on modular personalization, and we would be grateful if you would consider revisiting your assessment and raising your rating in light of the additional evidence.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions for Submission 13112

Dear Area Chair and Reviewers,

Thank you for your feedback. We have uploaded a revised manuscript with additional comparisons against TARA, PureCC, and FLUX.1-Kontext-dev; separate ablations of relative scaling and gating; and cross-backbone validation on SD1.5 and FLUX.1-dev (Secs. 4.2–4.5). V-Engram achieves the highest CLIP-I and DINOv2 in held-out contexts among the evaluated methods. On 20 matched two-subject prompts, Qwen3.5-Plus judges both identities preserved in 70.0% of assessments for V-Engram versus 5.0% for DreamBooth-LoRA. We also clarified the method description and updated Figure 1. Thank you for considering the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.