- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes V-Engram, an optimization method for subject-based customization for text-to-image diffusion models. The proposed method aims to introduce trigger words for the specialized concepts, which try to optimize a set of associated modulation parameters, which brings the ability of modifying both the represnetations in the text encoders of the diffusion model along with the MMDiT. The authors demonstrate the effectiveness of their method on Stable Diffusion 3.5, and compare with optimization based methods Textual Inversion and DreamBooth-LoRA. Overall, the authors show the effectiveness of their method uboth qualitatively and quantitatively, where they match the performance of DreamBooth-LoRA without intervening the model weights and surpass Textual Inversion. In addition, the authors also show the abilities if their method in terms of modelling detailed concepts such as human faces and generating multiple concepts in a given generation.
Critical strengths
- The proposed method offers a new optimization method for per-concept customization of text-to-image generation models, which proposes learned modulation artifacts. In presented qualitative and quantitative comparisons the method apper to be effective in reflecting the concepts in generation.
- Shwoing the flexibility of the method, the authors demonstrate the effectiveness of their approach on multiple concepts, and human identities, which are subjects with high level of detail.
- The paper provides the experiment setup in detail, which makes reproducibility possible.
Critical weaknesses
- The paper is mainly designed as a Stable Diffusion 3.5 oriented approach. While the concept of defining engrams for a trigger token can be generalized into multiple generators, the paper has no such demontsrations and the results are shown in a single model. Furthermore, the methodology appears to be a model specific approach, which reduces the claim of novelty in the proposed method.
- Overall, the definitions provided in Sec 3.2 are not coherent and makes understanding the method difficult. It appears that the method attempts to learn injectable activations per text encoder with the flow matching objective, however this is not straightforward to the reader in the papers current form where I, as a reviewer, was able to understand this with the help of Figure 2 and with the familiarity on how DreamBooth-LoRA is trained. Furhtermore, the paper needs a revision in terms of clarity.
- While the method compares with methods such as textual inversion and DreamBooth-LoRA, the comparisons miss a family of methods that use external adapters and image conditioning for personalization, which also serve as common workflows for customized content creations. Additionally, in the included comparisons, prompt following comparisons only cover DreamBooth-LoRA for comparison, which provides a narrow comparison with closely reported metrics. Additionally, to resolve the tension between the subject alignment and prompt alignment, the authors are advised to run user studies which can provide a better assessment for a joint objective, which is the main goal of personalization.
- This is not very critical but can be corrected in a revision: In Fig. 1, the captions seem off from the actual generations, which makes it harder to identify the strength of the method on prompt alignment. A presentation that aligns the text captions and the generations would benefit the presentation.
Questions for authors
- Is the method transferrable to other MMDiT based models (e.g. FLUX-1.dev), if not can the limitations be shown over examples or discussed?
- Can the authors provide comparisons with adaptor based methods, in addition to optimization based methods? Such comparisons can be better suited with identity specific generations (as presented in Figure 8 and Table 4), where concepts have more details and can demontsrate the V-engram's utilization of activation space better.
- For evaluations, can the authors provide user studies to resolve the tension between prompt following and subject following?
3 comments
Rebuttal Response by Reviewer TNUU
I appreciate the authors' detailed response to the reviewers' concerns and the additional experiments presented in the revised manuscript. From my perspective, the concerns regarding generalizability beyond Stable Diffusion 3.5, methodological clarity, limited baseline comparisons, and the caption inconsistencies have been substantially addressed. Considering these improvements, I am raising my score from weak rejection to weak acceptance.
However, I would like to highlight that the method still exhibits a trade-off between subject fidelity and prompt alignment. (reflected in the lower contextual CLIP-T score compared to other methods, see Table 1). Additionally, while I acknowledge that conducting a human evaluation within the limited rebuttal timeframe is challenging, I believe such a study would be a valuable addition to the final version, in case of acceptance, to better assess the balance between subject preservation and prompt following.
Thank You for Your Reassessment and Helpful Suggestions
Thank you for your thoughtful reassessment and for recognizing the improvements in the revised manuscript. We sincerely appreciate your constructive guidance throughout the discussion.
We agree that the trade-off between subject fidelity and prompt alignment remains important. In the latest revision, we have added Qwen assessments of identity preservation, prompt compliance, and joint success across the main baselines in Table 1, and revised the discussion to characterize this trade-off more explicitly.
We also appreciate your suggestion regarding human evaluation. Such a study would provide valuable complementary evidence, and we will carefully consider it for the final version, should the paper be accepted.
Thank you again for your time and helpful feedback.
Cross-Backbone Validation and Joint Composition Evaluation
Thank you again for your helpful feedback. We have uploaded a revised manuscript addressing the questions you raised.
We took your feedback seriously and carried out additional experiments to address the concerns you raised. We believe the new results make the value of V-Engram clearer: it allows subject memories to be stored independently, retrieved on demand, and used together in a single generation. If the revised manuscript addresses your main concerns, we would be grateful if you would consider raising your overall rating.