- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper presents UniFacet, a visual tokenizer for image generation and editing. It supplements a frozen visual encoder with a regularized residual branch to recover reconstruction-relevant details. The completed representation is used for source-image conditioning, while compact reconstruction and semantic facets form the generation target. Experiments evaluate reconstruction, semantic preservation, generation, and editing, with UniFacet-1.5B achieving 0.89 on GenEval and 7.47 on GEdit-Bench-EN.
Critical strengths
-
The design of separation between source-image conditioning and generation targets is reasonable , since source conditioning needs to preserve image details, whereas generation benefits from a compact target that is easier to model.
-
Table 6 provides a convincing comparison of the overall design. With the same backbone, training data, and training schedule, UniFacet improves GEdit-EN from 6.29 to 7.35 over the dual-stream baseline while halving the source-image token sequence. The reconstruction and understanding evaluations also support the feasibility of recovering image details while largely preserving semantic capabilities.
Critical weaknesses
-
Although the additional branch is called a semantic facet, its semantic content is not directly evaluated. The understanding and alignment experiments assess the high-dimensional unified representation rather than the learned 16-dimensional projection. Table 5 shows that the branch helps generation, but does not establish what semantic information it retains or how that information contributes to the improvement.
-
The benefit of jointly adapting the semantic projection is not sufficiently isolated. Table 5 lacks a fixed-projection baseline with the same dimensionality and training budget. The comparison in Appendix I addresses direct semantic alignment rather than this question. It therefore remains unclear whether joint adaptation is necessary or whether a fixed auxiliary representation would provide similar gains.
-
After learning the semantic projection in the preliminary 50k-step training run, the paper states that the projection is reused, but does not explain whether the generator weights are also carried over. If they are reused, the final generator has already received this additional training. The authors should clarify which generator checkpoint is used to start the subsequent multimodal training.
Questions for authors
-
What evidence shows that the learned 16-dimensional facet retains semantic information, and does joint adaptation outperform a fixed projection under matched dimensions and training budgets?
-
Are the generator weights from the preliminary 50k-step run reused for subsequent multimodal training?
4 comments
W2 & Q1: Is joint adaptation necessary, or would a fixed projection of the same dimension provide similar gains?
We first note that the semantic projection in UniFacet is also fixed during final training. It is adapted only during the first 30k steps of the preliminary run, and is then frozen together with its normalization statistics for all multimodal training of UniFacet-1.5B (Appendix F.2). The joint-adaptation step serves a single purpose: to learn adaptively a compact semantic projection whose features suit flow-matching generation. The question is therefore not whether the projection is fixed, but how its weights are obtained.
We considered two alternatives for obtaining this fixed projection:
- Fixed PCA projection. CoReDi reports that replacing the jointly learned projection with a fixed PCA projection degrades generation quality. PCA retains the directions of largest variance in the upstream features, which need not be the directions most useful for generative modeling.
- Projection learned with a separate semantic objective. Training the projection with another semantic objective (e.g., feature reconstruction or distillation) would require an additional training stage and data, and offers no guarantee that the resulting features suit generative modeling.
We therefore adopt the more natural option of learning the projection jointly with the generative model, following CoReDi. It selects the projection directly under the generative objective, at negligible cost: a single 1280→16 linear layer adapted for 30k steps.
Our main contribution is the semantic facet itself: it is derived from the unified space and generated jointly with the frozen reconstruction facet. The specific procedure used to obtain its projection is not part of this contribution. Table 5 and the probing results in our previous response support this design. We agree that a systematic comparison of projection strategies would be informative and will explore it in future work. We will also state explicitly in Sec. 3.1.2 that the adaptation strategy follows CoReDi.
W3 & Q2: Which generator checkpoint initializes the subsequent multimodal training?
Thank you for pointing out this ambiguity. No generator weights from the preliminary 50k-step run are carried over. The only transferred components are the semantic projector, a single linear layer, and its normalization statistics, both kept frozen. Curriculum phase C1 initializes the generator anew from Qwen2.5-1.5B-Instruct. The final generator therefore receives no training beyond the C1–C4 and SFT schedule in Table 8. We will state this initialization and parameter transfer explicitly in Appendix F.2.
We hope these responses address the reviewer's concerns, and we are happy to provide further clarification.
Dear Reviewer iXCi,
Thank you again for your careful review. With the discussion period drawing to a close, we would like to check whether our response has resolved your concerns. In it we added a direct probing analysis of the 16-d semantic facet, explained how its projection is obtained and used, and clarified that no generator weights are carried over from the preliminary run.
If anything remains unclear, or you would like further analysis, we would be glad to use the remaining time to provide it. Thank you for your time and consideration.
Best regards,
The Authors
Thank you for the detailed responses and for revising the unclear parts of the manuscript. My concerns have been addressed.
Response to Reviewer iXCi
We sincerely thank the reviewer for the careful reading and constructive questions. In response, we add a direct probing analysis of the semantic facet (W1), clarify how the semantic projection is obtained and used (W2), and specify the checkpoint used to start multimodal training (W3).
W1 & Q1: What evidence shows that the learned 16-dimensional facet retains semantic information?
We directly analyze the frozen 16-d semantic facet zsem used in UniFacet-1.5B, and compare it with the 64-d reconstruction facet zrec and the 1280-d VFM features zvfm.
Qualitative evidence. This figure compares per-image PCA visualizations of zvfm and zsem. Despite the compression to 16 dimensions, zsem keeps the object-level layout of the VFM features. Objects such as people, horses, the bird and the teddy bear form coherent regions with clear boundaries against the background.
Quantitative evidence. We fit linear foreground/background probes on an eight-breed Oxford-IIIT Pet subset (256/64/192 train/val/test images). The tokenizer and projections are frozen; only the linear readouts are trained. We report:
How this relates to the generation gain. The decoder consumes only zrec, so zsem cannot help by adding decoding capacity. Together with the probes, Table 5 indicates that the gain comes from exposing object-level structure that zrec lacks, rather than from adding latent channels: (i) the gain is insensitive to dsem (0.70–0.71 for 8/16/32 channels); and (ii) widening zrec from 64 to 128 channels lowers GenEval from 0.65 to 0.57, whereas adding a 16-d semantic facet raises it to 0.66.
We will add this analysis and the figures to the revision.